Urbanization_v8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0f2016b3",
   "metadata": {},
   "source": [
    "# Spatiotemporal Trends in Urbanization\n",
    "## Overview\n",
    "This repository investigates year-by-year change in cities and settlements in Central and Western Africa (CWA). The goal is to capture activity for every settlement locality in a country to produce indicators that are high frequency, spatially granular, and timely. The Jupyter Notebook is the primary script used to construct each country's dataset. It tracks population, built-area, and economic and climate indicators across a 16-year timeframe from 2000 to 2015. \n",
    "\n",
    "The repository is split into three sections: methodology and notebooks, source data, and outputs. Outputs are organized by country and include growth tables (\"urban panel datasets\"), charts, and country briefs.\n",
    "\n",
    "\n",
    "## Datasets\n",
    "Datasets used to create each African country's urban panel data are as follows:\n",
    "1. Most up-to-date administrative boundaries.\n",
    "2. City names: **UCDB, Africapolis, and GeoNames.**\n",
    "3. Settlement types: **GRID3 settlement extents.** Captured between 2009-2019.\n",
    "4. Built-up area, yearly: **World Settlement Footprint Evolution.** Resolution: 30m.\n",
    "5. Population, yearly: **WorldPop.** UN-adjusted, unconstrained. Resolution: 100m.\n",
    "6. Nighttime lights, yearly: **Harmonization of DMSP and VIIRS.** Resolution: 1km and 500m.\n",
    "7. Flood extents, by return period: **FATHOM.** Resolution: 90m.\n",
    "\n",
    "\n",
    "## Accessing Data\n",
    "Source data are available to the public by providers listed in the previous section, with the exception of flood data. Please note that the source data files in this repository have been fit for purpose and may not cover your area of interest. Some sources are also not global; GRID3 settlement extents are only available for sub-Saharan Africa, and Africapolis names for Africa.\n",
    "\n",
    "Results from the analysis are currently available for Cameroon and are under development for Central African Republic and four Sahel countries: Burkina Faso, Chad, Mali, and Niger. Results are available in the outputs folder by country. Please contact the CWA Geospatial team to inquire about new locations.\n",
    "<br>\n",
    "> **Walker Kosmidou-Bradley**, wkosmidoubradley@worldbank.org\n",
    "<br>\n",
    "> **Grace Doherty**, gdoherty2@worldbank.org\n",
    "\n",
    "## License\n",
    "Materials under this repository are open-source under an MIT license. The community is invited to test, adapt, and re-purpose materials as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85fd14a1",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9276b447",
   "metadata": {},
   "source": [
    "## 1. PREPARE WORKSPACE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0190f8",
   "metadata": {},
   "source": [
    "### 1.1 Off-script"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9093fd6",
   "metadata": {},
   "source": [
    "##### Off-script: Create folders in your working directory. (The folder where you are storing this script).\n",
    "> *ADM\n",
    "<br>Buildup\n",
    "<br>PlaceName\n",
    "<br>Population\n",
    "<br>Settlement\n",
    "<br>NTL*\n",
    "\n",
    "##### Before starting: Download datasets (as shapefile, GeoJSON, or tif where possible) and place or extract into corresponding folder. You can download the cleaned files from our [GitHub Repository](https://github.com/worldbank/Urban_Spatio_Temporal_Trends) or access original sources here:\n",
    "- ADM: *Varies by source.*\n",
    "- Buildup: https://download.geoservice.dlr.de/WSF_EVO/files/\n",
    "- PlaceName: \n",
    "    - GeoNames: (file: cities500.zip) https://download.geonames.org/export/dump/\n",
    "    - Africapolis: https://africapolis.org/en/data\n",
    "    - Urban Centres Database: https://ghsl.jrc.ec.europa.eu/ghs_stat_ucdb2015mt_r2019a.php\n",
    "- Population: https://hub.worldpop.org/geodata/listing?id=69\n",
    "- Settlement: https://data.grid3.org/datasets/GRID3::grid3-cameroon-settlement-extents-version-01-01-/explore\n",
    "- Nighttime Lights: https://eogdata.mines.edu/products/dmsp/#v4 and https://eogdata.mines.edu/products/vnl/#annual_v2\n",
    "\n",
    "##### Other off-script:\n",
    "- Convert GeoNames from .txt file to shape (delimiter = tab, header rows = 0) and rename fields.\n",
    "- If necessary, mosaic WSFE rasters that cover the area of interest to create a single file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b2a2548",
   "metadata": {},
   "source": [
    "### 1.2 Load all packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8c42795f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Built-in:\n",
    "# dir(), print(), range(), format(), int(), len(), list(), max(), min(), zip(), sorted(), sum(), open(), del, = None, try except, with as, for in, if elif else\n",
    "# Also: list.append(), list.insert(), list.remove(), count(), startswith(), endswith(), contains(), replace()\n",
    "\n",
    "import os, sys, glob, re, time, subprocess, string # os.getcwd(), os.path.join(), os.listdir(), os.remove(), time.ctime(), glob.glob(), string.zfill(), string.join()\n",
    "from os.path import exists # exists()\n",
    "from functools import reduce # reduce()\n",
    "\n",
    "import geopandas as gpd # read_file(), GeoDataFrame(), sjoin_nearest(), to_crs(), to_file(), .crs, buffer(), dissolve()\n",
    "import pandas as pd # .dtypes, Series(), concat(), DataFrame(), read_table(), merge(), to_csv(), .loc[], head(), sample(), astype(), unique(), rename(), between(), drop(), fillna(), idxmax(), isna(), isin(), apply(), info(), sort_values(), notna(), groupby(), value_counts(), duplicated(), drop_duplicates()\n",
    "from shapely.geometry import Point, LineString, Polygon, shape, MultiPoint\n",
    "from shapely.ops import cascaded_union\n",
    "from shapely.validation import make_valid  # in apply(make_valid)\n",
    "import shapely.wkt\n",
    "\n",
    "import numpy as np # median(), mean(), tolist(), .inf\n",
    "import fiona, rioxarray # fiona.open()\n",
    "import rasterio # open(), write_band(), .name, .count, .width, .height. nodatavals, .meta, update(), copy(), write()\n",
    "from rasterio.plot import show\n",
    "from rasterio import features # features.rasterize()\n",
    "from rasterio.features import shapes\n",
    "from rasterio import mask # rasterio.mask.mask()\n",
    "from osgeo import gdal, osr, ogr, gdal_array, gdalconst # Open(), SpatialReference, WarpOptions(), Warp(), GetDataTypeName(), GetRasterBand(), GetNoDataValue(), Translate(), GetProjection(), GetAttrValue()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5be5f9d7",
   "metadata": {},
   "source": [
    "### 1.3 Set up workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a20dfb25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\n",
      "Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Results\n"
     ]
    }
   ],
   "source": [
    "ProjectFolder = os.getcwd()\n",
    "ResultsFolder = os.path.join(ProjectFolder, 'Results')\n",
    "print(ProjectFolder)\n",
    "print(ResultsFolder)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "267db21b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ListFromRange(r1, r2):\n",
    "    return [item for item in range(r1, r2+1)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ff97aa01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1985, 1986, 1987, 1988, 1989, 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015] \n",
      "\n",
      " [1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015] \n",
      "\n",
      " [2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999]\n"
     ]
    }
   ],
   "source": [
    "WSFEYears = ListFromRange(1985, 2015) # All years in the WSFE dataset.\n",
    "AllStudyYears = ListFromRange(1999, 2015) # All years for which there will be growth stats in the present study.\n",
    "ReversedStudyYears = []\n",
    "for i in AllStudyYears:\n",
    "    ReversedStudyYears.insert(0,i)\n",
    "print(WSFEYears, '\\n\\n', AllStudyYears, '\\n\\n', ReversedStudyYears)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16bc3ea3",
   "metadata": {},
   "source": [
    "### 1.4 User-defined functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "317d1355",
   "metadata": {},
   "outputs": [],
   "source": [
    "# From Stack Exchange @RutgerH\n",
    "# https://gis.stackexchange.com/questions/163685/reclassify-a-raster-value-to-9999-and-set-it-to-the-nodata-value-using-python-a\n",
    "def readRaster(filename):\n",
    "    filehandle = gdal.Open(filename)\n",
    "    band1 = filehandle.GetRasterBand(1)\n",
    "    geotransform = filehandle.GetGeoTransform()\n",
    "    geoproj = filehandle.GetProjection()\n",
    "    Z = band1.ReadAsArray()\n",
    "    xsize = filehandle.RasterXSize\n",
    "    ysize = filehandle.RasterYSize\n",
    "    return xsize,ysize,geotransform,geoproj,Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "48d2e64f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Default arguments can be changed here, or can be specified below when running the functions.\n",
    "def writeRaster(filename,geotransform,geoprojection,data, NoDataVal=0):\n",
    "    (x,y) = data.shape\n",
    "    Dformat = \"GTiff\"\n",
    "    driver = gdal.GetDriverByName(Dformat)\n",
    "    # you can change the dataformat but be sure to be able to store negative values including -9999\n",
    "    dst_datatype = gdal.GDT_UInt32\n",
    "    dst_ds = driver.Create(filename,y,x,1,dst_datatype)\n",
    "    dst_ds.GetRasterBand(1).WriteArray(data)\n",
    "    dst_ds.SetGeoTransform(geotransform)\n",
    "    dst_ds.SetProjection(geoprojection)\n",
    "    dst_ds.GetRasterBand(1).SetNoDataValue(NoDataVal)\n",
    "    return 1\n",
    "    dst_ds = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ff2bd0dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Based on Stack Exchange @Kurt Schwehr:\n",
    "# https://stackoverflow.com/questions/10454316/how-to-project-and-resample-a-grid-to-match-another-grid-with-gdal-python\n",
    "def resampleRaster(InRaster_Path, MatchRaster_Path, OutFile_Path):\n",
    "    print('Loading for %s. %s' % (InRaster_Path, time.ctime()))\n",
    "    \n",
    "    RasterObject = gdal.Open(InRaster_Path)\n",
    "    In_proj = RasterObject.GetProjection()\n",
    "    [Match_x, Match_y, Match_geo, Match_proj, Match_Z] = readRaster(MatchRaster_Path)\n",
    "    print('---Specs to match to: \\n', \n",
    "      Match_proj, '\\n', Match_geo, '\\n', Match_x, '\\n', Match_y, '\\n')\n",
    "        \n",
    "    OutFile = gdal.GetDriverByName('GTiff').Create(OutFile_Path, Match_x, Match_y, 1, gdalconst.GDT_UInt32)\n",
    "    OutFile.SetGeoTransform(Match_geo)\n",
    "    OutFile.SetProjection(Match_proj)\n",
    "    print('---Created raster file for upsampled version. %s' % time.ctime())\n",
    "    \n",
    "    gdal.ReprojectImage(RasterObject, OutFile, In_proj, Match_proj, gdal.GRA_NearestNeighbour) # Nearest because categorical.\n",
    "    print('---Resampled flood values onto an empty raster matching the dimensions of the buildup layer. %s \\n\\n' % time.ctime())\n",
    "    \n",
    "    OutFile.GetRasterBand(1).SetNoDataValue(0)\n",
    "    \n",
    "    RasterObject = Outfile = None\n",
    "    return 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "65c5efde",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calcShell(A, B, OutFile, Calculation, OutType = ''):\n",
    "    \"\"\"Raster math using gdal_calc.py.\n",
    "\n",
    "    The OSgeo package for Python API does not make raster calculations\n",
    "    easy outside of the shell. This function plugs two raster files\n",
    "    into a string which subprocess.call() then commits to the terminal.\n",
    "    It only works with two objects but easy to adapt for 3 or more.\n",
    "\n",
    "        A : str\n",
    "            File path to the first raster for the calculation.\n",
    "        B : str\n",
    "            File path to the second raster for the calculation.\n",
    "        OutFile : str\n",
    "            File path where to store the raster generated from the calculation.\n",
    "        Calculation : str\n",
    "            Algebra that uses A and B to create a new raster. Use double quotes.\n",
    "    \"\"\"\n",
    "    print('Running for %s. %s' % (A, time.ctime()))\n",
    "    cmd = 'gdal_calc.py -A ' + A + ' -B ' + B + OutType + ' --outfile=' + OutFile + ' --overwrite --calc=' + Calculation\n",
    "    subprocess.call(cmd, shell=True)\n",
    "    cmd = A = B = None\n",
    "    print('Ran in shell. See OutFile folder to inspect results. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c7ae0d7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def RasterToShapefile(InRasterPath, OutFilePath = 'RastToShp.shp', Band=1, \n",
    "                      OutName='RastToShp', VariableName='value', Driver = 'ESRI Shapefile'):\n",
    "    \"\"\"Raster tiff to vector polygon shapefile.\n",
    "    \n",
    "    \"\"\"\n",
    "    Raster = gdal.Open(InRasterPath)\n",
    "    RasterBand = Raster.GetRasterBand(Band)\n",
    "    \n",
    "    OutDriver = ogr.GetDriverByName(Driver)\n",
    "    InProj = Raster.GetProjectionRef()\n",
    "    SpatRef = osr.SpatialReference()\n",
    "    SpatRef.ImportFromWkt(InProj)\n",
    "    print(InProj, '\\n\\n', SpatRef)\n",
    "    \n",
    "    OutFile = OutDriver.CreateDataSource(OutFilePath)\n",
    "    OutLayer = OutFile.CreateLayer(OutName, srs = SpatRef, geom_type = ogr.wkbPolygon)\n",
    "    OutField = ogr.FieldDefn(VariableName, ogr.OFTInteger)\n",
    "    OutLayer.CreateField(OutField)\n",
    "    OutField = OutLayer.GetLayerDefn().GetFieldIndex(VariableName)\n",
    "    print('\\n', OutFile, '\\n', OutLayer, '\\n', OutField)\n",
    "    \n",
    "    print('Vectorizing. Input: %s. %s' % (InRasterPath, time.ctime()))\n",
    "    gdal.Polygonize(RasterBand, None, OutLayer, 0, [], callback=None)\n",
    "    print('Completed polygons. Stored as: %s. %s' % (OutFilePath, time.ctime()))\n",
    "\n",
    "    del Raster, RasterBand, OutFile, OutLayer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d23cfdbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def rioStats(InRasterPath, Band = 1):\n",
    "    out = rasterio.open(InRasterPath)\n",
    "    stats = []\n",
    "    band = out.read(Band)\n",
    "    stats.append({\n",
    "        'raster': out.name,\n",
    "        'bands': out.count,\n",
    "        'data type': out.dtypes,\n",
    "        'no data value': out.nodatavals,\n",
    "        'width': out.width,\n",
    "        'height': out.height,\n",
    "        'min': band.min(),\n",
    "        'mean': band.mean(),\n",
    "        'median': np.median(band),\n",
    "        'max': band.max()})\n",
    "    print(\"\\n\", stats)\n",
    "    \n",
    "    out = band = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "95b9a04e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ShapeToRaster(Shapefile, ValueVar, MetaRasterPath, OutFilePath = 'ShpToRast.tif', Band=1):\n",
    "    \"\"\"\n",
    "    Polygon spatial object to raster tiff.\n",
    "    \"\"\"\n",
    "    # Copy and update the metadata from another raster for the output\n",
    "    MetaRaster = rasterio.open(MetaRasterPath)\n",
    "    meta = MetaRaster.meta.copy()\n",
    "    meta.update(compress='lzw')\n",
    "    MetaRaster.meta\n",
    "\n",
    "    print(\"Rasterizing dataset. %s\" % time.ctime())\n",
    "    with rasterio.open(OutFilePath, 'w+', **meta) as out:\n",
    "        out_arr = out.read(Band)\n",
    "\n",
    "        # this is where we create a generator of geom, value pairs to use in rasterizing\n",
    "        shapes = ((geom,value) for geom, value in zip(Shapefile.geometry, Shapefile[ValueVar]))\n",
    "\n",
    "        burned = features.rasterize(shapes=shapes, fill=0, out=out_arr, transform=out.transform)\n",
    "        out.write_band(1, burned)\n",
    "    out = burned = shapes = None\n",
    "    \n",
    "    print(\"Finished rasterizing. Checking contents. %s\" % time.ctime())\n",
    "    rioStats(OutFilePath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7ccda52e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def MaskByZone(MaskPath, SourceFolder, DestFolder, MaskLayerName = None, dstSRS = 'ESRI:102022'):\n",
    "    \"\"\"\n",
    "    Reduces the size of a raster's valid data cells to vector areas of interest.\n",
    "    This is useful if the raster data needs to be vectorized later to save space.\n",
    "    \n",
    "    The script prepares the vector zones as a list of geometries in the desired\n",
    "    spatial reference system, then warps each raster in the specified source\n",
    "    folder to the same SRS. Masking in rasterio then reclassifies any raster cells\n",
    "    falling outside of a mask polygon as NoData.\n",
    "    \"\"\"\n",
    "    \n",
    "    ProjSRS = osr.SpatialReference()\n",
    "    ProjSRS.SetFromUserInput(dstSRS)\n",
    "    ProjWarp = gdal.WarpOptions(dstSRS = dstSRS)\n",
    "    \n",
    "    AnnualizedSourceFiles = []\n",
    "    AnnualizedSourceFiles = AnnualizedSourceFiles + [i for i in os.listdir(''.join([SourceFolder, r'/'])) if i.endswith('tif')]\n",
    "    print(AnnualizedSourceFiles)\n",
    "    \n",
    "    \n",
    "    ### 1. ASSIGN SPATIAL REFERENCE SYSTEM OF VECTOR MASK AND LOAD GEOMETRIES\n",
    "    Vector = gpd.read_file(filename=MaskPath, layer=MaskLayerName)\n",
    "    if Vector.crs != dstSRS:\n",
    "        if MaskLayerName == None:\n",
    "            MaskPath = MaskPath + '_temp'\n",
    "        else:\n",
    "            MaskLayerName = MaskLayerName + '_temp'\n",
    "        Vector.to_crs(dstSRS).to_file(filename=MaskPath, layer=MaskLayerName)\n",
    "    Vector = None # We're reloading the geometries with fiona\n",
    "    \n",
    "    with fiona.open(MaskPath, mode=\"r\", layer=MaskLayerName) as Vector:\n",
    "        MaskGeom = [feature[\"geometry\"] for feature in Vector] # Identify the bounding areas of the mask.\n",
    "    \n",
    "    \n",
    "    ### 2. PREPARE DESTINATION FILES\n",
    "    for FileName in AnnualizedSourceFiles:\n",
    "    \n",
    "        InputRasterPath = os.path.join(ProjectFolder, SourceFolder, FileName)\n",
    "        \n",
    "        Sensor = re.search('[A-Z]+_', FileName)\n",
    "        if Sensor is None:\n",
    "            Sensor = ''\n",
    "        else:\n",
    "            Sensor = Sensor.group(0)\n",
    "\n",
    "        Year = re.search('\\d{4}', FileName)\n",
    "        if Year is None:\n",
    "            Year = ''\n",
    "        else:\n",
    "            Year = Year.group(0)\n",
    "\n",
    "        if FileName.endswith('avg.tif') == True:\n",
    "            IndicType = '_avg'\n",
    "        elif FileName.endswith('cfc.tif') == True:\n",
    "            IndicType = '_cfc'\n",
    "        else:\n",
    "            IndicType = ''\n",
    "\n",
    "        TempOutputName = 'Temp_' + Sensor + Year + IndicType + '.tif'\n",
    "        TempOutputPath = os.path.join(ProjectFolder, DestFolder, TempOutputName)\n",
    "        FinalOutputName = 'Msk_' + Sensor + Year + IndicType + '.tif'\n",
    "        FinalOutputPath = os.path.join(ProjectFolder, DestFolder, FinalOutputName)\n",
    "\n",
    "    ### 3. ASSIGN SPATIAL REFERENCE SYSTEM OF RASTER(S)\n",
    "        InputRasterObject = gdal.Open(InputRasterPath)\n",
    "        SourceSRS = osr.SpatialReference(wkt=InputRasterObject.GetProjection())\n",
    "        print('Source projection: ', SourceSRS.GetAttrValue('projcs'))\n",
    "        print('Destination projection: ', ProjSRS.GetAttrValue('projcs'))\n",
    "\n",
    "        if SourceSRS.GetAttrValue('projcs') != ProjSRS.GetAttrValue('projcs'):\n",
    "            Warp = gdal.Warp(TempOutputPath, # Where to store the warped raster\n",
    "                         InputRasterObject, # Which raster to warp\n",
    "                         format='GTiff', \n",
    "                         options=ProjWarp) # Reproject to Africa Albers Equal Area Conic\n",
    "            print('Finished gdal.Warp() for %s. %s \\n' % (YearFile, time.ctime()))\n",
    "\n",
    "            Warp = None # Close the files\n",
    "        else:\n",
    "            pass\n",
    "        InputRasterObject = None\n",
    "        \n",
    "    ### 4. RECLASSIFY AS NODATA IF OUTSIDE OF SETTLEMENT BUFFER ZONE.\n",
    "        if exists(TempOutputPath):\n",
    "            NewInputPath = TempOutputPath \n",
    "            print(\"We warped the data, so we'll use that file for next step.\")\n",
    "        else:\n",
    "            NewInputPath = InputRasterPath \n",
    "            print(\"We skipped the warp, so we continue to use the source file.\")\n",
    "\n",
    "        with rasterio.open(NewInputPath) as InputRasterObject:\n",
    "            MaskedOutputRaster, OutTransform = rasterio.mask.mask(\n",
    "                InputRasterObject, MaskGeom, crop=True) # Anything outside the mask is reclassed to the raster's NoData value.\n",
    "            OutMetaData = InputRasterObject.meta.copy()\n",
    "        print('Finished rasterio.mask.mask() for %s. %s \\n' % (FileName, time.ctime()))\n",
    "\n",
    "        OutMetaData.update({\"driver\": \"GTiff\",\n",
    "                         \"height\": MaskedOutputRaster.shape[1],\n",
    "                         \"width\": MaskedOutputRaster.shape[2],\n",
    "                         \"transform\": OutTransform})\n",
    "\n",
    "        with rasterio.open(FinalOutputPath, \"w\", **OutMetaData) as dest:\n",
    "            dest.write(MaskedOutputRaster)\n",
    "        print('Written to file. %s \\n' % time.ctime())\n",
    "        InputRasterObject = None\n",
    "\n",
    "        if exists(TempOutputPath):\n",
    "            try:  # Finally, remove the intermediate file from disk\n",
    "                os.remove(TempOutputPath)\n",
    "            except OSError:\n",
    "                pass\n",
    "            print('Removed intermediate file. %s \\n' % time.ctime())\n",
    "        else:\n",
    "            pass\n",
    "\n",
    "\n",
    "    print('\\n \\n Finished all years in list. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "55a3e947",
   "metadata": {},
   "outputs": [],
   "source": [
    "def BatchZonalStats(FolderName, Zones, \n",
    "                    CRS = 'ESRI:102022', \n",
    "                    JoinField = 'Sett_ID',\n",
    "                    StatsWanted = ['count', 'sum', 'mean', 'max', 'min'],\n",
    "                    SeriesStart = 1999, SeriesEnd = 2015, \n",
    "                    AnnualizedFiles = None, VarPrefix = None):\n",
    "    \"\"\"\n",
    "    Normally, we would use numpy to generate a point gdf from the raster's matrix. \n",
    "    However, I was running into a lot of memory errors with that method.\n",
    "    This method uses some extra steps: tif to xyz to df to gdf. But it saves to file\n",
    "    and deletes intermediate files along the way, circumventing memory issues.\n",
    "    \n",
    "    Run MaskByZone() prior to reduce the raster to only your area(s) of interest.\n",
    "    \n",
    "    \"\"\"\n",
    "    if AnnualizedFiles is None:\n",
    "        AnnualizedFiles = [i for i in os.listdir(FolderName) if i.endswith('.tif')]\n",
    "    print(AnnualizedFiles)\n",
    "    AllSummaries = pd.DataFrame(Zones).drop(columns='geometry')[[JoinField]]\n",
    "    print(AllSummaries)\n",
    "    \n",
    "    if VarPrefix is None:\n",
    "        VarPrefix = FolderName[:3].upper()\n",
    "    \n",
    "    for FileName in AnnualizedFiles:\n",
    "    ### STEP 1: TIF TO XYZ ###\n",
    "        print('Loading data for %s. %s \\n' % (FileName, time.ctime()))\n",
    "        \n",
    "        Sensor = re.search('[A-Z]+_', FileName)\n",
    "        if Sensor is None:\n",
    "            Sensor = ''\n",
    "        else:\n",
    "            Sensor = Sensor.group(0)\n",
    "            \n",
    "        Year = re.search('\\d{4}', FileName)\n",
    "        if Year is None:\n",
    "            Year = ''\n",
    "        else:\n",
    "            Year = Year.group(0)\n",
    "        \n",
    "        InputRasterPath = os.path.join(ProjectFolder, FolderName, FileName)\n",
    "        InputRasterObject = gdal.Open(InputRasterPath)\n",
    "        XYZOutputPath = FolderName + r'/{}'.format(\n",
    "            FileName.replace('.tif', '.xyz')) # New file path will be the same as original, but .tif is replaced with .xyz\n",
    "\n",
    "        # Create an .xyz version of the .tif\n",
    "        if exists(XYZOutputPath):\n",
    "            print(\"Already created xyz file.\")\n",
    "        else:\n",
    "            print(\"Creating XYZ (gdal.Translate()).\")\n",
    "            XYZ = gdal.Translate(XYZOutputPath, # Specify a destination path\n",
    "                                 InputRasterObject, # Input is the masked .tif file\n",
    "                                 format='XYZ', \n",
    "                                 creationOptions=[\"ADD_HEADER_LINE=YES\"])\n",
    "            print('Finished gdal.Translate() for year %s. %s \\n' % (Year, time.ctime()))\n",
    "            XYZ = None # Reload XYZ as a point geodataframe\n",
    "\n",
    "        InputRasterObject = None\n",
    "\n",
    "\n",
    "    ### STEP 2: GENERATE GEODATAFRAME WITH JOIN FIELD ###\n",
    "        InputXYZ = pd.read_table(XYZOutputPath, delim_whitespace=True)\n",
    "        InputXYZ = InputXYZ.loc[InputXYZ['Z'] > 0] # Subset to only the features that have a value.\n",
    "        \n",
    "        if re.search('WSFE', FileName) is not None: # Scale back up to years if working with flood/building data.\n",
    "            InputXYZ['Z'] = InputXYZ['Z'] + 1900\n",
    "            \n",
    "        print('Loaded XYZ file as a pandas dataframe. %s \\n' % time.ctime())\n",
    "        ValObject = gpd.GeoDataFrame(InputXYZ,\n",
    "                                     geometry = gpd.points_from_xy(InputXYZ['X'], InputXYZ['Y']),\n",
    "                                     crs = CRS)\n",
    "        print('Created geodataframe from non-NoData points. %s \\n' % time.ctime())\n",
    "        del InputXYZ\n",
    "\n",
    "        # Sjoin_nearest: No need to group by ADM this time. \n",
    "        ValObject_withID = pd.DataFrame(gpd.sjoin_nearest(ValObject, \n",
    "                                        Zones, \n",
    "                                        how='left')).drop(columns='geometry')[['Z', JoinField]] # No need for max_distance parameter this time. We've already narrowed down to nearby raster cells.\n",
    "\n",
    "        print('\\nJoined zone ID onto vectorized raster cells. %s \\n' % time.ctime())\n",
    "        print(ValObject_withID.sample(10))\n",
    "        del ValObject\n",
    "\n",
    "        ValObject_withID.to_csv(''.join([FolderName, r'/', FileName.replace('.tif', '.csv')]))\n",
    "        print('\\nExported as table. %s \\n' % time.ctime())\n",
    "\n",
    "#         # Remove the temporary xyz file.\n",
    "#         try:  \n",
    "#             os.remove(os.path.join(XYZOutputPath))\n",
    "#         except OSError:\n",
    "#             pass\n",
    "#         print('Removed (or skipped if error) intermediate xyz file. %s \\n' % time.ctime())\n",
    "\n",
    "\n",
    "    ### STEP 3: AGGREGATE BY SETTLEMENT AND MERGE ONTO SUMMARIES TABLE ###\n",
    "        GroupedVals = ValObject_withID[ValObject_withID['Z'].notna()].groupby(JoinField, as_index=False)\n",
    "        \n",
    "        # Run this block if the variable is about cloud-free coverage.\n",
    "        if re.search('cfc', FileName) is not None:\n",
    "            VariableName = ''.join([VarPrefix, 'cfc_', Sensor, Year])\n",
    "            AllSummaries = AllSummaries.merge(GroupedVals.mean().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            print('\\nCount of cloud-free observations averaged to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "            \n",
    "            # Save in-progress results\n",
    "            AllSummaries.to_csv(os.path.join(ResultsFolder, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "            print(AllSummaries.sample(10))\n",
    "        \n",
    "        # Run this block if we're working with the flooded buildings data.\n",
    "        elif re.search('WSFE', FileName) is not None:\n",
    "            for BuiltYear in AllStudyYears:\n",
    "                Grouped_Subset = GroupedVals[GroupedVals['Z'].between(\n",
    "                    1985, BuiltYear, inclusive=True)] # Inclusive parameter means we include the years 1985 and the named year.\n",
    "                if 'count' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'ct', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.count().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'sum' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'sum', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.sum().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'mean' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'avg', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.mean().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'max' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'max', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.max().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                if 'min' in StatsWanted:\n",
    "                    VariableName = ''.join([VarPrefix, 'min', Sensor, BuiltYear])\n",
    "                    AllSummaries = AllSummaries.merge(GroupedVals.min().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "                print('\\nDesired aggregation methods applied to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "\n",
    "                # Save in-progress results\n",
    "                AllSummaries.to_csv(os.path.join(ResultsFolder, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "                print(AllSummaries.sample(10))\n",
    "        \n",
    "        # Anything else takes the standard aggregation method.\n",
    "        else:\n",
    "            if 'count' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'ct', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.count().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'sum' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'sum', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.sum().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'mean' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'avg', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.mean().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'max' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'max', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.max().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            if 'min' in StatsWanted:\n",
    "                VariableName = ''.join([VarPrefix, 'min', Sensor, Year])\n",
    "                AllSummaries = AllSummaries.merge(GroupedVals.min().rename(columns={'Z': VariableName}), how = 'left', on=JoinField)\n",
    "            print('\\nDesired aggregation methods applied to settlement level, year %s. %s \\n' % (Year, time.ctime()))\n",
    "            \n",
    "            # Save in-progress results\n",
    "            AllSummaries.to_csv(os.path.join(ResultsFolder, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "            print(AllSummaries.sample(10))\n",
    "\n",
    "    \n",
    "    print('\\n\\nFinished. All years masked and assigned their nearest settlement. %s' % time.ctime())\n",
    "    print(AllSummaries.sample(10))\n",
    "    AllSummaries.to_csv(os.path.join(ResultsFolder, ''.join([VarPrefix, '%sto%s.csv' % (SeriesStart, SeriesEnd)])))\n",
    "    print('Saved to file. %s \\n' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1584def0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5537c1ce",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c08550",
   "metadata": {},
   "source": [
    "## 2. PREPARE BUILDUP, SETTLEMENT, AND ADMIN DATASETS\n",
    "Projection for all datasets: Africa Albers Equal Area Conic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e591ea5f",
   "metadata": {},
   "source": [
    "### 2.1 Prepare raster locations for GRID3 and Admin areas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e7946e25",
   "metadata": {},
   "outputs": [],
   "source": [
    "ADM_vec = gpd.read_file(glob.glob('ADM/*.shp')[0])[['geometry']].to_crs(\"ESRI:102022\") # This glob() function pulls the first file ([0]) in the ADM folder which ended in '.shp'\n",
    "GRID3_vec = gpd.read_file(glob.glob('Settlement/*.shp')[0])[['type','geometry']].to_crs(\"ESRI:102022\")\n",
    "ADM_vec['ADM_ID'] = range(0,len(ADM_vec))\n",
    "ADM_vec['ADM_ID'] = ADM_vec['ADM_ID'] + 1 # We have to add 1 if we want our rasterized version's NoData value to be 0. Otherwise the first feature won't be valid.\n",
    "GRID3_vec['G3_ID'] = range(0,len(GRID3_vec))\n",
    "GRID3_vec['G3_ID'] = GRID3_vec['G3_ID'] +1\n",
    "ADM_vec.to_file(driver='GPKG', filename=r'ADM/ADM_equalarea.gpkg', layer='ADM')\n",
    "GRID3_vec.to_file(driver='GPKG', filename=r'Settlement/Settlement_equalarea.gpkg', layer='GRID3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d44280f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 351 entries, 0 to 350\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   ADM_ID    351 non-null    int64   \n",
      " 1   geometry  351 non-null    geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 5.6 KB\n",
      "None \n",
      "\n",
      "      ADM_ID                                           geometry\n",
      "128     129  POLYGON ((-2814706.700 1366314.825, -2814686.8...\n",
      "196     197  POLYGON ((-3121153.823 1262207.751, -3120940.4...\n",
      "82       83  MULTIPOLYGON (((-3039069.415 1467544.605, -303...\n",
      "197     198  POLYGON ((-2651759.240 1596949.597, -2651690.5...\n",
      "308     309  POLYGON ((-2778815.854 1482839.071, -2778612.6... PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]] \n",
      "\n",
      " 3\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 615220 entries, 0 to 615219\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   type      615220 non-null  object  \n",
      " 1   G3_ID     615220 non-null  int64   \n",
      " 2   geometry  615220 non-null  geometry\n",
      "dtypes: geometry(1), int64(1), object(1)\n",
      "memory usage: 14.1+ MB\n",
      "None \n",
      "\n",
      "           type   G3_ID                                           geometry\n",
      "465989  Hamlet  465990  POLYGON ((-2893162.607 1608919.115, -2893120.7...\n",
      "147765  Hamlet  147766  POLYGON ((-2971008.407 1409142.044, -2970966.9...\n",
      "486803  Hamlet  486804  POLYGON ((-2765853.067 1617627.478, -2765811.2...\n",
      "136173  Hamlet  136174  POLYGON ((-2945499.809 1376213.954, -2945458.3...\n",
      "20140   Hamlet   20141  POLYGON ((-3061234.567 1204868.095, -3061223.6... PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]] \n",
      "\n",
      " 6\n"
     ]
    }
   ],
   "source": [
    "ADM_vec = gpd.read_file(r'ADM/ADM_equalarea.gpkg', layer='ADM')\n",
    "GRID3_vec = gpd.read_file(r'Settlement/Settlement_equalarea.gpkg', layer='GRID3')\n",
    "\n",
    "# We need to know how many digits need to be allocated to each dataset in the \"join\" serial.\n",
    "len_ADM = len(str(ADM_vec['ADM_ID'].max()))\n",
    "len_G3 =  len(str(GRID3_vec['G3_ID'].max()))\n",
    "\n",
    "print(ADM_vec.info(), \"\\n\\n\", \n",
    "      ADM_vec.sample(5),\n",
    "      ADM_vec.crs, \"\\n\\n\", \n",
    "      len_ADM) \n",
    "print(GRID3_vec.info(), \"\\n\\n\",\n",
    "      GRID3_vec.sample(5),\n",
    "      GRID3_vec.crs, \"\\n\\n\", \n",
    "      len_G3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca195d70",
   "metadata": {},
   "source": [
    "### 2.2 Reproject WSFE to project CRS, and ensure valid data range is only 1985-2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "44102e36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote the reclassed raster to file. Wed Jan 25 16:06:18 2023\n"
     ]
    }
   ],
   "source": [
    "InPath = glob.glob('Buildup/*.tif')[0]\n",
    "OutWGS = os.path.join(ProjectFolder, 'Buildup', 'WSFE_reclass.tif')\n",
    "\n",
    "# Together, x and y define the data's \"shape\".\n",
    "# geotransform contains the parameters detailing how the raster should be stretched and aligned.\n",
    "# geoproj is the map projection\n",
    "# Z are the values in the raster band.\n",
    "[xsize,ysize,geotransform,geoproj,Z] = readRaster(InPath)\n",
    "Z[Z<1985] = 0\n",
    "Z[Z>2015] = 0\n",
    "\n",
    "writeRaster(OutWGS,geotransform,geoproj,Z)\n",
    "print('Wrote the reclassed raster to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "71ebd2eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wrote the reclassed and reprojected raster to file. Wed Jan 25 16:07:25 2023\n"
     ]
    }
   ],
   "source": [
    "OutEqArea = os.path.join(ProjectFolder, 'Buildup', 'WSFE_equalarea.tif')\n",
    "\n",
    "# Whenever we want to work in a projected CRS, we'll use Africa Albers Equal Area Conic.\n",
    "ProjEqArea = gdal.WarpOptions(dstSRS='ESRI:102022')\n",
    "Warp = gdal.Warp(OutEqArea, # Where to store the warped raster\n",
    "                 OutWGS, # Which raster to warp\n",
    "                 format='GTiff', \n",
    "                 options=ProjEqArea)\n",
    "print('Wrote the reclassed and reprojected raster to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "695d3cd7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/BurkinaFaso/Buildup/WSFE_reclass.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (0.0,), 'width': 29405, 'height': 21083, 'min': 0, 'mean': 6.654020696638043, 'median': 0.0, 'max': 2015}]\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/BurkinaFaso/Buildup/WSFE_equalarea.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (0.0,), 'width': 28597, 'height': 22808, 'min': 0, 'mean': 6.506819179191691, 'median': 0.0, 'max': 2015}]\n"
     ]
    }
   ],
   "source": [
    "rioStats(OutWGS)\n",
    "rioStats(OutEqArea)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "de7dbcaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "InPath = Warp = OutWGS = OutEqArea = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b44be4f",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "440aa2a2",
   "metadata": {},
   "source": [
    "## 3. WSFE AND ADM; GRID3 AND ADM\n",
    "RASTERIZE: Bring ADM and GRID3 into raster space.\n",
    "\n",
    "RASTER MATH: \"Join\" ADM ID onto GRID3 and onto WSFE by creating unique concatenation string.\n",
    "\n",
    "VECTORIZE: Bring joined data into vector space.\n",
    "\n",
    "VECTOR MATH: Split unique ID from raster math step into separate columns."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9663bd1",
   "metadata": {},
   "source": [
    "### 3.1 Rasterize admin areas and GRID3 using WSFE specs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "af504c4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rasterizing dataset. Wed Jan 25 19:02:35 2023\n",
      "Finished rasterizing. Checking contents. Wed Jan 25 19:03:24 2023\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/BurkinaFaso/ADM/ADM_rasterized.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (0.0,), 'width': 28597, 'height': 22808, 'min': 0, 'mean': 84.24833096349128, 'median': 0.0, 'max': 351}]\n",
      "Rasterizing dataset. Wed Jan 25 19:03:42 2023\n",
      "Finished rasterizing. Checking contents. Wed Jan 25 19:07:57 2023\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/BurkinaFaso/Settlement/GRID3_rasterized.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (0.0,), 'width': 28597, 'height': 22808, 'min': 0, 'mean': 12265.64792682353, 'median': 0.0, 'max': 615220}]\n"
     ]
    }
   ],
   "source": [
    "# Copy and update the metadata from WSFE for the output\n",
    "WSFE = os.path.join(ProjectFolder, 'Buildup', 'WSFE_equalarea.tif')\n",
    "ADM_out = os.path.join(ProjectFolder, 'ADM', 'ADM_rasterized.tif')\n",
    "GRID3_out = os.path.join(ProjectFolder, 'Settlement', 'GRID3_rasterized.tif')\n",
    "\n",
    "ShapeToRaster(Shapefile=ADM_vec, ValueVar=\"ADM_ID\", MetaRasterPath=WSFE, OutFilePath=ADM_out)\n",
    "ShapeToRaster(GRID3_vec, \"G3_ID\", WSFE, GRID3_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "406ceaa3",
   "metadata": {},
   "source": [
    "### 3.2 Raster math to \"join\" admin to GRID3 and to WSFE.\n",
    "Processing is more rapid when \"joining,\" i.e. creating serial codes out of two datasets, in raster rather than vector space.\n",
    "Here, we are concatenating the ID fields of the two datasets to create a serial number that we can then split in vector space later to create two ID fields."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3e831cd9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running for Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Settlement\\GRID3_rasterized.tif. Wed Jan 25 19:08:11 2023\n",
      "Ran in shell. See OutFile folder to inspect results. Wed Jan 25 19:09:52 2023\n",
      "Running for Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Buildup\\WSFE_equalarea.tif. Wed Jan 25 19:09:52 2023\n",
      "Ran in shell. See OutFile folder to inspect results. Wed Jan 25 19:11:36 2023\n"
     ]
    }
   ],
   "source": [
    "# In paths\n",
    "InG3 = os.path.join(ProjectFolder, 'Settlement', 'GRID3_rasterized.tif')\n",
    "InWSFE = os.path.join(ProjectFolder, 'Buildup', 'WSFE_equalarea.tif')\n",
    "InADM = os.path.join(ProjectFolder, 'ADM', 'ADM_rasterized.tif')\n",
    "\n",
    "# Out paths\n",
    "G3_ADM = os.path.join(ProjectFolder, 'Settlement', 'GRID3_ADM.tif')\n",
    "WSFE_ADM = os.path.join(ProjectFolder, 'Buildup', 'WSFE_ADM.tif')\n",
    "\n",
    "# Calculations\n",
    "Calc = \"(A*\" + str(10**len_ADM) + \")+B\" \n",
    "# The number of digits in the largest ADM index value (len_ADM) is \n",
    "# the number of zeroes we tack onto the first variable in the serial.\n",
    "\n",
    "calcShell(A=InG3, B=InADM, OutFile=G3_ADM, Calculation=Calc)\n",
    "calcShell(A=InWSFE, B=InADM, OutFile=WSFE_ADM, Calculation=Calc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e46a2406",
   "metadata": {},
   "source": [
    "*Adding together the values to create join IDs. This is in effect a concatenation of their ID strings, by way of summation. The number of zeros in the calc multiplication corresponds with number of digits of the maximum value in the \"B\" dataset. (e.g. Chad ADM codes go up 4 digits, so it's calc=(A*10000)+B).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b8df5020",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/BurkinaFaso/Settlement/GRID3_ADM.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (4294967293.0,), 'width': 28597, 'height': 22808, 'min': 1188, 'mean': 4132712942.105214, 'median': 4294967293.0, 'max': 4294967293}]\n",
      "\n",
      " [{'raster': 'Q:/GIS/povertyequity/urban_growth/BurkinaFaso/Buildup/WSFE_ADM.tif', 'bands': 1, 'data type': ('uint32',), 'no data value': (4294967293.0,), 'width': 28597, 'height': 22808, 'min': 1985001, 'mean': 4281025071.9005694, 'median': 4294967293.0, 'max': 4294967293}]\n"
     ]
    }
   ],
   "source": [
    "rioStats(G3_ADM)\n",
    "rioStats(WSFE_ADM)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ea3639",
   "metadata": {},
   "source": [
    "### 3.3 Vectorize serialized layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "85dfc892",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]] \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",\n",
      "    GEOGCS[\"WGS 84\",\n",
      "        DATUM[\"WGS_1984\",\n",
      "            SPHEROID[\"WGS 84\",6378137,298.257223563,\n",
      "                AUTHORITY[\"EPSG\",\"7030\"]],\n",
      "            AUTHORITY[\"EPSG\",\"6326\"]],\n",
      "        PRIMEM[\"Greenwich\",0],\n",
      "        UNIT[\"degree\",0.0174532925199433,\n",
      "            AUTHORITY[\"EPSG\",\"9122\"]],\n",
      "        AUTHORITY[\"EPSG\",\"4326\"]],\n",
      "    PROJECTION[\"Albers_Conic_Equal_Area\"],\n",
      "    PARAMETER[\"latitude_of_center\",0],\n",
      "    PARAMETER[\"longitude_of_center\",25],\n",
      "    PARAMETER[\"standard_parallel_1\",20],\n",
      "    PARAMETER[\"standard_parallel_2\",-23],\n",
      "    PARAMETER[\"false_easting\",0],\n",
      "    PARAMETER[\"false_northing\",0],\n",
      "    UNIT[\"metre\",1,\n",
      "        AUTHORITY[\"EPSG\",\"9001\"]],\n",
      "    AXIS[\"Easting\",EAST],\n",
      "    AXIS[\"Northing\",NORTH]]\n",
      "\n",
      " <osgeo.ogr.DataSource; proxy of <Swig Object of type 'OGRDataSourceShadow *' at 0x0000018CAAEBFED0> > \n",
      " <osgeo.ogr.Layer; proxy of <Swig Object of type 'OGRLayerShadow *' at 0x0000018CAAEBED60> > \n",
      " 0\n",
      "Vectorizing. Input: Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Settlement\\GRID3_ADM.tif. Wed Jan 25 19:30:51 2023\n",
      "Completed polygons. Stored as: Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Settlement\\GRID3_ADM.shp. Wed Jan 25 23:18:31 2023\n",
      "PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH]] \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",\n",
      "    GEOGCS[\"WGS 84\",\n",
      "        DATUM[\"WGS_1984\",\n",
      "            SPHEROID[\"WGS 84\",6378137,298.257223563,\n",
      "                AUTHORITY[\"EPSG\",\"7030\"]],\n",
      "            AUTHORITY[\"EPSG\",\"6326\"]],\n",
      "        PRIMEM[\"Greenwich\",0],\n",
      "        UNIT[\"degree\",0.0174532925199433,\n",
      "            AUTHORITY[\"EPSG\",\"9122\"]],\n",
      "        AUTHORITY[\"EPSG\",\"4326\"]],\n",
      "    PROJECTION[\"Albers_Conic_Equal_Area\"],\n",
      "    PARAMETER[\"latitude_of_center\",0],\n",
      "    PARAMETER[\"longitude_of_center\",25],\n",
      "    PARAMETER[\"standard_parallel_1\",20],\n",
      "    PARAMETER[\"standard_parallel_2\",-23],\n",
      "    PARAMETER[\"false_easting\",0],\n",
      "    PARAMETER[\"false_northing\",0],\n",
      "    UNIT[\"metre\",1,\n",
      "        AUTHORITY[\"EPSG\",\"9001\"]],\n",
      "    AXIS[\"Easting\",EAST],\n",
      "    AXIS[\"Northing\",NORTH]]\n",
      "\n",
      " <osgeo.ogr.DataSource; proxy of <Swig Object of type 'OGRDataSourceShadow *' at 0x0000018CAAEBFE10> > \n",
      " <osgeo.ogr.Layer; proxy of <Swig Object of type 'OGRLayerShadow *' at 0x0000018CAAEBFED0> > \n",
      " 0\n",
      "Vectorizing. Input: Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Buildup\\WSFE_ADM.tif. Wed Jan 25 23:18:32 2023\n",
      "Completed polygons. Stored as: Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Buildup\\WSFE_ADM.shp. Wed Jan 25 23:21:10 2023\n"
     ]
    }
   ],
   "source": [
    "G3_in = os.path.join(ProjectFolder, 'Settlement', 'GRID3_ADM.tif')\n",
    "G3_out = os.path.join(ProjectFolder, 'Settlement', 'GRID3_ADM.shp')\n",
    "WSFE_in = os.path.join(ProjectFolder, 'Buildup', 'WSFE_ADM.tif')\n",
    "WSFE_out = os.path.join(ProjectFolder, 'Buildup', 'WSFE_ADM.shp')\n",
    "\n",
    "RasterToShapefile(G3_in, G3_out, OutName='GRID3_ADM', VariableName='gridcode', Driver = 'ESRI Shapefile')\n",
    "RasterToShapefile(WSFE_in, WSFE_out, OutName='WSFE_ADM', VariableName='gridcode', Driver = 'ESRI Shapefile')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03937b47",
   "metadata": {},
   "source": [
    "### 3.4 Vector math to split raster strings into admin area, GRID3, and WSFE year assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "bb900b13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 651922 entries, 0 to 651921\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   gridcode  651922 non-null  int64   \n",
      " 1   geometry  651922 non-null  geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 9.9 MB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 922083 entries, 0 to 922082\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   gridcode  922083 non-null  int64   \n",
      " 1   geometry  922083 non-null  geometry\n",
      "dtypes: geometry(1), int64(1)\n",
      "memory usage: 14.1 MB\n",
      "None \n",
      "\n",
      "           gridcode                                           geometry\n",
      "496831   279505220  POLYGON ((-2616149.075 1326384.485, -2616061.6...\n",
      "204663  2147483647  POLYGON ((-3010491.859 1478617.650, -3010433.5...\n",
      "437005   198318009  POLYGON ((-2800537.196 1362096.018, -2800449.7...\n",
      "544353   248887327  POLYGON ((-2679438.658 1300905.399, -2679380.3...\n",
      "608980    35049181  POLYGON ((-3159926.406 1230998.251, -3159838.9...\n",
      "503250   106471116  POLYGON ((-2987140.889 1322565.537, -2987053.4...\n",
      "488251   114288042  POLYGON ((-2963906.528 1331398.676, -2963819.0...\n",
      "283676   234606255  POLYGON ((-2745177.031 1434539.414, -2745118.7...\n",
      "624335    11974302  POLYGON ((-3080923.749 1206073.058, -3080836.2...\n",
      "379346   146263258  POLYGON ((-2947668.713 1389674.067, -2947610.4... \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]] \n",
      "\n",
      " None \n",
      "\n",
      "         gridcode                                           geometry\n",
      "906595   2007209  POLYGON ((-3116635.282 1192313.186, -3116606.1...\n",
      "568577   1985292  POLYGON ((-2411616.734 1384776.486, -2411558.4...\n",
      "245714   2004325  POLYGON ((-2933209.186 1453284.325, -2933180.0...\n",
      "118266   1997115  POLYGON ((-2775612.003 1529225.994, -2775582.8...\n",
      "864009   2000274  POLYGON ((-3045620.347 1242076.114, -3045591.1...\n",
      "75499    1994150  POLYGON ((-2784270.228 1556774.891, -2784241.0...\n",
      "232733   2002336  POLYGON ((-2743515.352 1461651.027, -2743486.1...\n",
      "663828   2001165  POLYGON ((-3106723.509 1346587.009, -3106694.3...\n",
      "900115   1994096  POLYGON ((-2936386.784 1199630.406, -2936357.6...\n",
      "219066   1999070  POLYGON ((-2879977.137 1467977.070, -2879947.9... \n",
      "\n",
      " PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]] \n",
      "\n",
      " 2147483647 2147483647\n"
     ]
    }
   ],
   "source": [
    "# Load newly created vectorized datasets.\n",
    "GRID3_ADM = gpd.read_file(r\"Settlement/GRID3_ADM.shp\")\n",
    "WSFE_ADM = gpd.read_file(r\"Buildup/WSFE_ADM.shp\")\n",
    "print(GRID3_ADM.info(), \"\\n\\n\", GRID3_ADM.sample(10), \"\\n\\n\", GRID3_ADM.crs, \"\\n\\n\", \n",
    "      WSFE_ADM.info(), \"\\n\\n\", WSFE_ADM.sample(10), \"\\n\\n\", WSFE_ADM.crs, \"\\n\\n\", \n",
    "      GRID3_ADM['gridcode'].max(), WSFE_ADM['gridcode'].max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8f41b97c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         gridcode                                           geometry  \\\n",
      "358880  300874343  POLYGON ((-2663434.060 1398565.510, -2663346.6...   \n",
      "300619  125123293  POLYGON ((-3062820.188 1425443.905, -3062761.8...   \n",
      "298126  151834064  POLYGON ((-2967200.735 1426784.910, -2967113.2...   \n",
      "215761  538380031  POLYGON ((-2593118.780 1471796.018, -2593031.3...   \n",
      "347848  301186343  POLYGON ((-2653872.115 1403550.549, -2653842.9...   \n",
      "8714    500320166  POLYGON ((-2727918.886 1699475.263, -2727831.4...   \n",
      "417300  291808144  POLYGON ((-2581166.349 1372503.379, -2581137.1...   \n",
      "506974  106117116  POLYGON ((-2974663.717 1320524.878, -2974576.2...   \n",
      "144825  544916038  POLYGON ((-2630113.013 1524066.042, -2630083.8...   \n",
      "90412   578743270  POLYGON ((-2560847.215 1576831.654, -2560730.6...   \n",
      "\n",
      "       gridstring  Sett_ID  ADM_ID  \n",
      "358880  300874343   300874     343  \n",
      "300619  125123293   125123     293  \n",
      "298126  151834064   151834      64  \n",
      "215761  538380031   538380      31  \n",
      "347848  301186343   301186     343  \n",
      "8714    500320166   500320     166  \n",
      "417300  291808144   291808     144  \n",
      "506974  106117116   106117     116  \n",
      "144825  544916038   544916      38  \n",
      "90412   578743270   578743     270           gridcode                                           geometry  \\\n",
      "52246    2000219  POLYGON ((-2859045.806 1575286.584, -2859016.6...   \n",
      "791574   1994037  POLYGON ((-3049818.274 1294462.747, -3049759.9...   \n",
      "685491   2003307  POLYGON ((-2839513.784 1336442.019, -2839484.6...   \n",
      "423773   1985218  POLYGON ((-2772084.578 1427542.869, -2772055.4...   \n",
      "22009    2013161  POLYGON ((-2869919.603 1611989.294, -2869890.4...   \n",
      "657969   2010028  POLYGON ((-2990872.380 1349560.541, -2990843.2...   \n",
      "154443   2014038  POLYGON ((-2622591.727 1510043.799, -2622562.5...   \n",
      "698160   2005116  POLYGON ((-2973905.758 1331223.762, -2973876.6...   \n",
      "70427    2013225  POLYGON ((-2850270.972 1560214.859, -2850241.8...   \n",
      "436726   2004249  POLYGON ((-2652210.436 1425473.058, -2652181.2...   \n",
      "\n",
      "       gridstring  year  ADM_ID  \n",
      "52246     2000219  2000     219  \n",
      "791574    1994037  1994      37  \n",
      "685491    2003307  2003     307  \n",
      "423773    1985218  1985     218  \n",
      "22009     2013161  2013     161  \n",
      "657969    2010028  2010      28  \n",
      "154443    2014038  2014      38  \n",
      "698160    2005116  2005     116  \n",
      "70427     2013225  2013     225  \n",
      "436726    2004249  2004     249  \n"
     ]
    }
   ],
   "source": [
    "# Split serial back into separate dataset fields.\n",
    "# For example, Burkina: WSFE and ADM: 4+3=7 digits. GRID3 and ADM: 6+3=9 digits.\n",
    "\n",
    "G3_Fill = len_G3 + len_ADM\n",
    "WSFE_Fill = 4 + len_ADM\n",
    "\n",
    "GRID3_ADM['gridstring'] = GRID3_ADM['gridcode'].astype(str).str.zfill(G3_Fill)\n",
    "WSFE_ADM['gridstring'] = WSFE_ADM['gridcode'].astype(str).str.zfill(WSFE_Fill)\n",
    "\n",
    "GRID3_ADM['Sett_ID'] = GRID3_ADM['gridstring'].str[:-len_ADM].astype(int) # Remove the last 3 digits to get the GRID3 portion.\n",
    "GRID3_ADM['ADM_ID'] = GRID3_ADM['gridstring'].str[-len_ADM:].astype(int) # Keep only the last 3 digits to get the ADM portion.\n",
    "WSFE_ADM['year'] = WSFE_ADM['gridstring'].str[:-len_ADM].astype(int)\n",
    "WSFE_ADM['ADM_ID'] = WSFE_ADM['gridstring'].str[-len_ADM:].astype(int)\n",
    "\n",
    "print(GRID3_ADM.sample(10), WSFE_ADM.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "3127bbf9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Thu Jan 26 09:47:24 2023\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 620198 entries, 0 to 620197\n",
      "Data columns (total 5 columns):\n",
      " #   Column      Non-Null Count   Dtype   \n",
      "---  ------      --------------   -----   \n",
      " 0   Sett_ID     620198 non-null  int64   \n",
      " 1   ADM_ID      620198 non-null  int64   \n",
      " 2   geometry    620198 non-null  geometry\n",
      " 3   gridcode    620198 non-null  int64   \n",
      " 4   gridstring  620198 non-null  object  \n",
      "dtypes: geometry(1), int64(3), object(1)\n",
      "memory usage: 23.7+ MB\n",
      "None    Sett_ID  ADM_ID                                           geometry  \\\n",
      "0        1     188  POLYGON ((-3057776.844 1150654.589, -3057718.5...   \n",
      "1        2     188  POLYGON ((-3062499.513 1164531.070, -3062412.0...   \n",
      "2        3     209  POLYGON ((-3116256.302 1193537.581, -3115993.9...   \n",
      "3        4     284  POLYGON ((-3124885.375 1215635.003, -3124710.4...   \n",
      "4        5      14  POLYGON ((-3102204.907 1222602.396, -3102088.2...   \n",
      "\n",
      "   gridcode gridstring  \n",
      "0      1188  000001188  \n",
      "1      2188  000002188  \n",
      "2      3209  000003209  \n",
      "3      4284  000004284  \n",
      "4      5014  000005014   \n",
      "\n",
      " Thu Jan 26 09:59:35 2023\n"
     ]
    }
   ],
   "source": [
    "# Dissolve any features that have the same G3 and ADM values so that we have a single unique feature per settlement.\n",
    "# Note: we do NOT want to dissolve the WSFE features. Distinct features for noncontiguous builtup areas of the same year is necessary to separate them in the Near tool step.\n",
    "print(time.ctime())\n",
    "GRID3_ADM = GRID3_ADM.dissolve(by=['Sett_ID', 'ADM_ID'], as_index=False)\n",
    "print(GRID3_ADM.info(), GRID3_ADM.head(), \"\\n\\n\", time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f172cebe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: WSFE (922083, 5) and GRID3 (620198, 5)\n",
      "\n",
      "After: WSFE (922083, 5) and GRID3 (620198, 5)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Remove features where year, settlement, or admin area = 0.\n",
    "# This was supposed to be resolved earlier with the gdal_calc NoDataValue parameter.\n",
    "\n",
    "print(\"Before: WSFE %s and GRID3 %s\\n\" % (WSFE_ADM.shape, GRID3_ADM.shape))\n",
    "WSFE_ADM = WSFE_ADM.loc[(WSFE_ADM[\"year\"] != 0) & (WSFE_ADM[\"ADM_ID\"] != 0)] # Since we change the datatype to integer, no need to include all digits. Otherwise, it would need to be: != '0000'\n",
    "GRID3_ADM = GRID3_ADM.loc[(GRID3_ADM[\"Sett_ID\"] != 0) & (GRID3_ADM[\"ADM_ID\"] != 0)]\n",
    "print(\"After: WSFE %s and GRID3 %s\\n\" % (WSFE_ADM.shape, GRID3_ADM.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "9c6f651d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The Bounded_ID is our new unique settlement identifier for subsequent matching steps.\n",
    "GRID3_ADM['Bounded_ID'] = GRID3_ADM.index\n",
    "WSFE_ADM['WSFE_ID'] = WSFE_ADM.index\n",
    "GRID3_ADM = GRID3_ADM[['Sett_ID', 'Bounded_ID', 'ADM_ID', 'geometry']]\n",
    "WSFE_ADM = WSFE_ADM[['WSFE_ID', 'year', 'ADM_ID', 'geometry']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "35ec5fdb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "620198 620198 922083 10026\n"
     ]
    }
   ],
   "source": [
    "# Validation: \n",
    "# The first two printed numbers should be the same. There shouldn't be any GRID3 rows with matching Sett_ID and ADM_IDs.\n",
    "# The latter two numbers should be different, and the first should be larger. We never dissolved WSFE by any column.\n",
    "\n",
    "print(len(GRID3_ADM[['Sett_ID', 'ADM_ID']]),\n",
    "      len(GRID3_ADM[['Sett_ID', 'ADM_ID']].drop_duplicates()),\n",
    "      len(WSFE_ADM[['year', 'ADM_ID']]),\n",
    "      len(WSFE_ADM[['year', 'ADM_ID']].drop_duplicates()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "82419540",
   "metadata": {},
   "outputs": [],
   "source": [
    "GRID3_ADM.to_file(\n",
    "    driver='GPKG', filename='Settlement/GRID3_ADM.gpkg', layer='GRID3_ADM_cleaned')\n",
    "WSFE_ADM.to_file(\n",
    "    driver='GPKG', filename=r'Buildup/WSFE_ADM.gpkg', layer='WSFE_ADM_cleaned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5abd7f5e",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "013f0a8e",
   "metadata": {},
   "source": [
    "## 4. UNIQUE SETTLEMENTS FROM WSFE AND GRID3: TWO VERSIONS\n",
    "\n",
    "Note that there are 2 versions here, so that we can create a fragmentation index:\n",
    "1. **Boundless, aka boundary-agnostic settlements**: Unique settlements are linked to GRID3 settlement IDs. Administrative areas do not influence the extents of the settlement.\n",
    "2. **Bounded, aka politically-defined settlements**: Settlements in the Boundless dataset which spread across more than one administrative area are split into separate settlements in the Bounded dataset. The largest polygon after the split is considered the \"principal\" settlement, and polygons in other admin areas are considered \"fragments.\" By dividing the fragment area(s) of the Bounded settlement by the area of the Boundless settlement, we can acquire a fragmentation index for each locality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32df6624",
   "metadata": {},
   "source": [
    "### 4.1 BOUNDED SETTLEMENTS: Near Join by ADM group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8d294c3a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of admin areas with GRID3 features: 352\n",
      "Number of admin areas with WSFE features: 352\n",
      "Number of admin areas where one dataset is observed but the other is not: 0\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of admin areas with GRID3 features: %s\" % len(GRID3_ADM['ADM_ID'].unique().tolist()))\n",
    "print(\"Number of admin areas with WSFE features: %s\" % len(WSFE_ADM['ADM_ID'].unique().tolist()))\n",
    "print(\"Number of admin areas where one dataset is observed but the other is not: %s\" % (\n",
    "    len(GRID3_ADM['ADM_ID'].unique().tolist()) - len(WSFE_ADM['ADM_ID'].unique().tolist())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "74e1c2f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 2,\n",
       " 3,\n",
       " 4,\n",
       " 5,\n",
       " 6,\n",
       " 7,\n",
       " 8,\n",
       " 9,\n",
       " 10,\n",
       " 11,\n",
       " 12,\n",
       " 13,\n",
       " 14,\n",
       " 15,\n",
       " 16,\n",
       " 17,\n",
       " 18,\n",
       " 19,\n",
       " 20,\n",
       " 21,\n",
       " 22,\n",
       " 23,\n",
       " 24,\n",
       " 25,\n",
       " 26,\n",
       " 27,\n",
       " 28,\n",
       " 29,\n",
       " 30,\n",
       " 31,\n",
       " 32,\n",
       " 33,\n",
       " 34,\n",
       " 35,\n",
       " 36,\n",
       " 37,\n",
       " 38,\n",
       " 39,\n",
       " 40,\n",
       " 41,\n",
       " 42,\n",
       " 43,\n",
       " 44,\n",
       " 45,\n",
       " 46,\n",
       " 47,\n",
       " 48,\n",
       " 49,\n",
       " 50,\n",
       " 51,\n",
       " 52,\n",
       " 53,\n",
       " 54,\n",
       " 55,\n",
       " 56,\n",
       " 57,\n",
       " 58,\n",
       " 59,\n",
       " 60,\n",
       " 61,\n",
       " 62,\n",
       " 63,\n",
       " 64,\n",
       " 65,\n",
       " 66,\n",
       " 67,\n",
       " 68,\n",
       " 69,\n",
       " 70,\n",
       " 71,\n",
       " 72,\n",
       " 73,\n",
       " 74,\n",
       " 75,\n",
       " 76,\n",
       " 77,\n",
       " 78,\n",
       " 79,\n",
       " 80,\n",
       " 81,\n",
       " 82,\n",
       " 83,\n",
       " 84,\n",
       " 85,\n",
       " 86,\n",
       " 87,\n",
       " 88,\n",
       " 89,\n",
       " 90,\n",
       " 91,\n",
       " 92,\n",
       " 93,\n",
       " 94,\n",
       " 95,\n",
       " 96,\n",
       " 97,\n",
       " 98,\n",
       " 99,\n",
       " 100,\n",
       " 101,\n",
       " 102,\n",
       " 103,\n",
       " 104,\n",
       " 105,\n",
       " 106,\n",
       " 107,\n",
       " 108,\n",
       " 109,\n",
       " 110,\n",
       " 111,\n",
       " 112,\n",
       " 113,\n",
       " 114,\n",
       " 115,\n",
       " 116,\n",
       " 117,\n",
       " 118,\n",
       " 119,\n",
       " 120,\n",
       " 121,\n",
       " 122,\n",
       " 123,\n",
       " 124,\n",
       " 125,\n",
       " 126,\n",
       " 127,\n",
       " 128,\n",
       " 129,\n",
       " 130,\n",
       " 131,\n",
       " 132,\n",
       " 133,\n",
       " 134,\n",
       " 135,\n",
       " 136,\n",
       " 137,\n",
       " 138,\n",
       " 139,\n",
       " 140,\n",
       " 141,\n",
       " 142,\n",
       " 143,\n",
       " 144,\n",
       " 145,\n",
       " 146,\n",
       " 147,\n",
       " 148,\n",
       " 149,\n",
       " 150,\n",
       " 151,\n",
       " 152,\n",
       " 153,\n",
       " 154,\n",
       " 155,\n",
       " 156,\n",
       " 157,\n",
       " 158,\n",
       " 159,\n",
       " 160,\n",
       " 161,\n",
       " 162,\n",
       " 163,\n",
       " 164,\n",
       " 165,\n",
       " 166,\n",
       " 167,\n",
       " 168,\n",
       " 169,\n",
       " 170,\n",
       " 171,\n",
       " 172,\n",
       " 173,\n",
       " 174,\n",
       " 175,\n",
       " 176,\n",
       " 177,\n",
       " 178,\n",
       " 179,\n",
       " 180,\n",
       " 181,\n",
       " 182,\n",
       " 183,\n",
       " 184,\n",
       " 185,\n",
       " 186,\n",
       " 187,\n",
       " 188,\n",
       " 189,\n",
       " 190,\n",
       " 191,\n",
       " 192,\n",
       " 193,\n",
       " 194,\n",
       " 195,\n",
       " 196,\n",
       " 197,\n",
       " 198,\n",
       " 199,\n",
       " 200,\n",
       " 201,\n",
       " 202,\n",
       " 203,\n",
       " 204,\n",
       " 205,\n",
       " 206,\n",
       " 207,\n",
       " 208,\n",
       " 209,\n",
       " 210,\n",
       " 211,\n",
       " 212,\n",
       " 213,\n",
       " 214,\n",
       " 215,\n",
       " 216,\n",
       " 217,\n",
       " 218,\n",
       " 219,\n",
       " 220,\n",
       " 221,\n",
       " 222,\n",
       " 223,\n",
       " 224,\n",
       " 225,\n",
       " 226,\n",
       " 227,\n",
       " 228,\n",
       " 229,\n",
       " 230,\n",
       " 231,\n",
       " 232,\n",
       " 233,\n",
       " 234,\n",
       " 235,\n",
       " 236,\n",
       " 237,\n",
       " 238,\n",
       " 239,\n",
       " 240,\n",
       " 241,\n",
       " 242,\n",
       " 243,\n",
       " 244,\n",
       " 245,\n",
       " 246,\n",
       " 247,\n",
       " 248,\n",
       " 249,\n",
       " 250,\n",
       " 251,\n",
       " 252,\n",
       " 253,\n",
       " 254,\n",
       " 255,\n",
       " 256,\n",
       " 257,\n",
       " 258,\n",
       " 259,\n",
       " 260,\n",
       " 261,\n",
       " 262,\n",
       " 263,\n",
       " 264,\n",
       " 265,\n",
       " 266,\n",
       " 267,\n",
       " 268,\n",
       " 269,\n",
       " 270,\n",
       " 271,\n",
       " 272,\n",
       " 273,\n",
       " 274,\n",
       " 275,\n",
       " 276,\n",
       " 277,\n",
       " 278,\n",
       " 279,\n",
       " 280,\n",
       " 281,\n",
       " 282,\n",
       " 283,\n",
       " 284,\n",
       " 285,\n",
       " 286,\n",
       " 287,\n",
       " 288,\n",
       " 289,\n",
       " 290,\n",
       " 291,\n",
       " 292,\n",
       " 293,\n",
       " 294,\n",
       " 295,\n",
       " 296,\n",
       " 297,\n",
       " 298,\n",
       " 299,\n",
       " 300,\n",
       " 301,\n",
       " 302,\n",
       " 303,\n",
       " 304,\n",
       " 305,\n",
       " 306,\n",
       " 307,\n",
       " 308,\n",
       " 309,\n",
       " 310,\n",
       " 311,\n",
       " 312,\n",
       " 313,\n",
       " 314,\n",
       " 315,\n",
       " 316,\n",
       " 317,\n",
       " 318,\n",
       " 319,\n",
       " 320,\n",
       " 321,\n",
       " 322,\n",
       " 323,\n",
       " 324,\n",
       " 325,\n",
       " 326,\n",
       " 327,\n",
       " 328,\n",
       " 329,\n",
       " 330,\n",
       " 331,\n",
       " 332,\n",
       " 333,\n",
       " 334,\n",
       " 335,\n",
       " 336,\n",
       " 337,\n",
       " 338,\n",
       " 339,\n",
       " 340,\n",
       " 341,\n",
       " 342,\n",
       " 343,\n",
       " 344,\n",
       " 345,\n",
       " 346,\n",
       " 347,\n",
       " 348,\n",
       " 349,\n",
       " 350,\n",
       " 351,\n",
       " 647]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ADM_IDs = sorted(GRID3_ADM['ADM_ID'].unique().tolist())\n",
    "ADM_IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "72693d07",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We're creating this field to help in removing duplicates from the sjoin_nearest, next section.\n",
    "GRID3_ADM['G3_Area'] = GRID3_ADM['geometry'].area / 10**6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "f3bafaff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 0 entries\n",
      "Data columns (total 6 columns):\n",
      " #   Column      Non-Null Count  Dtype   \n",
      "---  ------      --------------  -----   \n",
      " 0   Sett_ID     0 non-null      int64   \n",
      " 1   Bounded_ID  0 non-null      int64   \n",
      " 2   ADM_ID      0 non-null      int64   \n",
      " 3   geometry    0 non-null      geometry\n",
      " 4   G3_Area     0 non-null      float64 \n",
      " 5   year        0 non-null      int32   \n",
      "dtypes: float64(1), geometry(1), int32(1), int64(3)\n",
      "memory usage: 0.0 bytes\n"
     ]
    }
   ],
   "source": [
    "# Create empty geodataframe to append onto using the dataframe whose geometry we want to retain.\n",
    "Bounded = GRID3_ADM[0:0]\n",
    "Bounded[\"year\"] = pd.Series(dtype='int')\n",
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1b40bc28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed near join in admin area 1. Thu Jan 26 10:05:30 2023 \n",
      "\n",
      "Completed near join in admin area 2. Thu Jan 26 10:05:30 2023 \n",
      "\n",
      "Completed near join in admin area 3. Thu Jan 26 10:05:30 2023 \n",
      "\n",
      "Completed near join in admin area 4. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 5. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 6. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 7. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 8. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 9. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 10. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 11. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 12. Thu Jan 26 10:05:31 2023 \n",
      "\n",
      "Completed near join in admin area 13. Thu Jan 26 10:05:32 2023 \n",
      "\n",
      "Completed near join in admin area 14. Thu Jan 26 10:05:32 2023 \n",
      "\n",
      "Completed near join in admin area 15. Thu Jan 26 10:05:32 2023 \n",
      "\n",
      "Completed near join in admin area 16. Thu Jan 26 10:05:32 2023 \n",
      "\n",
      "Completed near join in admin area 17. Thu Jan 26 10:05:32 2023 \n",
      "\n",
      "Completed near join in admin area 18. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 19. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 20. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 21. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 22. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 23. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 24. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 25. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 26. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 27. Thu Jan 26 10:05:33 2023 \n",
      "\n",
      "Completed near join in admin area 28. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 29. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 30. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 31. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 32. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 33. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 34. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 35. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 36. Thu Jan 26 10:05:34 2023 \n",
      "\n",
      "Completed near join in admin area 37. Thu Jan 26 10:05:38 2023 \n",
      "\n",
      "Completed near join in admin area 38. Thu Jan 26 10:05:38 2023 \n",
      "\n",
      "Completed near join in admin area 39. Thu Jan 26 10:05:38 2023 \n",
      "\n",
      "Completed near join in admin area 40. Thu Jan 26 10:05:38 2023 \n",
      "\n",
      "Completed near join in admin area 41. Thu Jan 26 10:05:38 2023 \n",
      "\n",
      "Completed near join in admin area 42. Thu Jan 26 10:05:38 2023 \n",
      "\n",
      "Completed near join in admin area 43. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 44. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 45. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 46. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 47. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 48. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 49. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 50. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 51. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 52. Thu Jan 26 10:05:39 2023 \n",
      "\n",
      "Completed near join in admin area 53. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 54. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 55. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 56. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 57. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 58. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 59. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 60. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 61. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 62. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 63. Thu Jan 26 10:05:40 2023 \n",
      "\n",
      "Completed near join in admin area 64. Thu Jan 26 10:05:41 2023 \n",
      "\n",
      "Completed near join in admin area 65. Thu Jan 26 10:05:41 2023 \n",
      "\n",
      "Completed near join in admin area 66. Thu Jan 26 10:05:41 2023 \n",
      "\n",
      "Completed near join in admin area 67. Thu Jan 26 10:05:41 2023 \n",
      "\n",
      "Completed near join in admin area 68. Thu Jan 26 10:05:42 2023 \n",
      "\n",
      "Completed near join in admin area 69. Thu Jan 26 10:05:42 2023 \n",
      "\n",
      "Completed near join in admin area 70. Thu Jan 26 10:05:42 2023 \n",
      "\n",
      "Completed near join in admin area 71. Thu Jan 26 10:05:42 2023 \n",
      "\n",
      "Completed near join in admin area 72. Thu Jan 26 10:05:42 2023 \n",
      "\n",
      "Completed near join in admin area 73. Thu Jan 26 10:05:42 2023 \n",
      "\n",
      "Completed near join in admin area 74. Thu Jan 26 10:05:42 2023 \n",
      "\n",
      "Completed near join in admin area 75. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 76. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 77. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 78. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 79. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 80. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 81. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 82. Thu Jan 26 10:05:43 2023 \n",
      "\n",
      "Completed near join in admin area 83. Thu Jan 26 10:05:44 2023 \n",
      "\n",
      "Completed near join in admin area 84. Thu Jan 26 10:05:44 2023 \n",
      "\n",
      "Completed near join in admin area 85. Thu Jan 26 10:05:44 2023 \n",
      "\n",
      "Completed near join in admin area 86. Thu Jan 26 10:05:44 2023 \n",
      "\n",
      "Completed near join in admin area 87. Thu Jan 26 10:05:45 2023 \n",
      "\n",
      "Completed near join in admin area 88. Thu Jan 26 10:05:45 2023 \n",
      "\n",
      "Completed near join in admin area 89. Thu Jan 26 10:05:45 2023 \n",
      "\n",
      "Completed near join in admin area 90. Thu Jan 26 10:05:45 2023 \n",
      "\n",
      "Completed near join in admin area 91. Thu Jan 26 10:05:45 2023 \n",
      "\n",
      "Completed near join in admin area 92. Thu Jan 26 10:05:45 2023 \n",
      "\n",
      "Completed near join in admin area 93. Thu Jan 26 10:05:46 2023 \n",
      "\n",
      "Completed near join in admin area 94. Thu Jan 26 10:05:46 2023 \n",
      "\n",
      "Completed near join in admin area 95. Thu Jan 26 10:05:46 2023 \n",
      "\n",
      "Completed near join in admin area 96. Thu Jan 26 10:05:46 2023 \n",
      "\n",
      "Completed near join in admin area 97. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 98. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 99. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 100. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 101. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 102. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 103. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 104. Thu Jan 26 10:05:47 2023 \n",
      "\n",
      "Completed near join in admin area 105. Thu Jan 26 10:05:48 2023 \n",
      "\n",
      "Completed near join in admin area 106. Thu Jan 26 10:05:48 2023 \n",
      "\n",
      "Completed near join in admin area 107. Thu Jan 26 10:05:48 2023 \n",
      "\n",
      "Completed near join in admin area 108. Thu Jan 26 10:05:48 2023 \n",
      "\n",
      "Completed near join in admin area 109. Thu Jan 26 10:05:48 2023 \n",
      "\n",
      "Completed near join in admin area 110. Thu Jan 26 10:05:48 2023 \n",
      "\n",
      "Completed near join in admin area 111. Thu Jan 26 10:05:49 2023 \n",
      "\n",
      "Completed near join in admin area 112. Thu Jan 26 10:05:49 2023 \n",
      "\n",
      "Completed near join in admin area 113. Thu Jan 26 10:05:49 2023 \n",
      "\n",
      "Completed near join in admin area 114. Thu Jan 26 10:05:49 2023 \n",
      "\n",
      "Completed near join in admin area 115. Thu Jan 26 10:05:49 2023 \n",
      "\n",
      "Completed near join in admin area 116. Thu Jan 26 10:05:49 2023 \n",
      "\n",
      "Completed near join in admin area 117. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 118. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 119. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 120. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 121. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 122. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 123. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 124. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 125. Thu Jan 26 10:05:50 2023 \n",
      "\n",
      "Completed near join in admin area 126. Thu Jan 26 10:05:50 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed near join in admin area 127. Thu Jan 26 10:05:51 2023 \n",
      "\n",
      "Completed near join in admin area 128. Thu Jan 26 10:05:51 2023 \n",
      "\n",
      "Completed near join in admin area 129. Thu Jan 26 10:05:51 2023 \n",
      "\n",
      "Completed near join in admin area 130. Thu Jan 26 10:05:51 2023 \n",
      "\n",
      "Completed near join in admin area 131. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 132. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 133. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 134. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 135. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 136. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 137. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 138. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 139. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 140. Thu Jan 26 10:05:52 2023 \n",
      "\n",
      "Completed near join in admin area 141. Thu Jan 26 10:05:53 2023 \n",
      "\n",
      "Completed near join in admin area 142. Thu Jan 26 10:05:53 2023 \n",
      "\n",
      "Completed near join in admin area 143. Thu Jan 26 10:05:53 2023 \n",
      "\n",
      "Completed near join in admin area 144. Thu Jan 26 10:05:53 2023 \n",
      "\n",
      "Completed near join in admin area 145. Thu Jan 26 10:05:53 2023 \n",
      "\n",
      "Completed near join in admin area 146. Thu Jan 26 10:05:53 2023 \n",
      "\n",
      "Completed near join in admin area 147. Thu Jan 26 10:05:54 2023 \n",
      "\n",
      "Completed near join in admin area 148. Thu Jan 26 10:05:54 2023 \n",
      "\n",
      "Completed near join in admin area 149. Thu Jan 26 10:05:54 2023 \n",
      "\n",
      "Completed near join in admin area 150. Thu Jan 26 10:05:54 2023 \n",
      "\n",
      "Completed near join in admin area 151. Thu Jan 26 10:05:54 2023 \n",
      "\n",
      "Completed near join in admin area 152. Thu Jan 26 10:05:54 2023 \n",
      "\n",
      "Completed near join in admin area 153. Thu Jan 26 10:05:55 2023 \n",
      "\n",
      "Completed near join in admin area 154. Thu Jan 26 10:05:55 2023 \n",
      "\n",
      "Completed near join in admin area 155. Thu Jan 26 10:05:55 2023 \n",
      "\n",
      "Completed near join in admin area 156. Thu Jan 26 10:05:55 2023 \n",
      "\n",
      "Completed near join in admin area 157. Thu Jan 26 10:05:57 2023 \n",
      "\n",
      "Completed near join in admin area 158. Thu Jan 26 10:05:57 2023 \n",
      "\n",
      "Completed near join in admin area 159. Thu Jan 26 10:05:57 2023 \n",
      "\n",
      "Completed near join in admin area 160. Thu Jan 26 10:05:57 2023 \n",
      "\n",
      "Completed near join in admin area 161. Thu Jan 26 10:05:57 2023 \n",
      "\n",
      "Completed near join in admin area 162. Thu Jan 26 10:05:58 2023 \n",
      "\n",
      "Completed near join in admin area 163. Thu Jan 26 10:05:58 2023 \n",
      "\n",
      "Completed near join in admin area 164. Thu Jan 26 10:05:58 2023 \n",
      "\n",
      "Completed near join in admin area 165. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 166. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 167. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 168. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 169. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 170. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 171. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 172. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 173. Thu Jan 26 10:05:59 2023 \n",
      "\n",
      "Completed near join in admin area 174. Thu Jan 26 10:06:00 2023 \n",
      "\n",
      "Completed near join in admin area 175. Thu Jan 26 10:06:00 2023 \n",
      "\n",
      "Completed near join in admin area 176. Thu Jan 26 10:06:00 2023 \n",
      "\n",
      "Completed near join in admin area 177. Thu Jan 26 10:06:00 2023 \n",
      "\n",
      "Completed near join in admin area 178. Thu Jan 26 10:06:00 2023 \n",
      "\n",
      "Completed near join in admin area 179. Thu Jan 26 10:06:00 2023 \n",
      "\n",
      "Completed near join in admin area 180. Thu Jan 26 10:06:01 2023 \n",
      "\n",
      "Completed near join in admin area 181. Thu Jan 26 10:06:01 2023 \n",
      "\n",
      "Completed near join in admin area 182. Thu Jan 26 10:06:01 2023 \n",
      "\n",
      "Completed near join in admin area 183. Thu Jan 26 10:06:01 2023 \n",
      "\n",
      "Completed near join in admin area 184. Thu Jan 26 10:06:01 2023 \n",
      "\n",
      "Completed near join in admin area 185. Thu Jan 26 10:06:01 2023 \n",
      "\n",
      "Completed near join in admin area 186. Thu Jan 26 10:06:01 2023 \n",
      "\n",
      "Completed near join in admin area 187. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 188. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 189. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 190. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 191. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 192. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 193. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 194. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 195. Thu Jan 26 10:06:02 2023 \n",
      "\n",
      "Completed near join in admin area 196. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 197. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 198. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 199. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 200. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 201. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 202. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 203. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 204. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 205. Thu Jan 26 10:06:03 2023 \n",
      "\n",
      "Completed near join in admin area 206. Thu Jan 26 10:06:04 2023 \n",
      "\n",
      "Completed near join in admin area 207. Thu Jan 26 10:06:04 2023 \n",
      "\n",
      "Completed near join in admin area 208. Thu Jan 26 10:06:04 2023 \n",
      "\n",
      "Completed near join in admin area 209. Thu Jan 26 10:06:04 2023 \n",
      "\n",
      "Completed near join in admin area 210. Thu Jan 26 10:06:04 2023 \n",
      "\n",
      "Completed near join in admin area 211. Thu Jan 26 10:06:04 2023 \n",
      "\n",
      "Completed near join in admin area 212. Thu Jan 26 10:06:05 2023 \n",
      "\n",
      "Completed near join in admin area 213. Thu Jan 26 10:06:05 2023 \n",
      "\n",
      "Completed near join in admin area 214. Thu Jan 26 10:06:05 2023 \n",
      "\n",
      "Completed near join in admin area 215. Thu Jan 26 10:06:05 2023 \n",
      "\n",
      "Completed near join in admin area 216. Thu Jan 26 10:06:05 2023 \n",
      "\n",
      "Completed near join in admin area 217. Thu Jan 26 10:06:05 2023 \n",
      "\n",
      "Completed near join in admin area 218. Thu Jan 26 10:06:19 2023 \n",
      "\n",
      "Completed near join in admin area 219. Thu Jan 26 10:06:20 2023 \n",
      "\n",
      "Completed near join in admin area 220. Thu Jan 26 10:06:20 2023 \n",
      "\n",
      "Completed near join in admin area 221. Thu Jan 26 10:06:20 2023 \n",
      "\n",
      "Completed near join in admin area 222. Thu Jan 26 10:06:20 2023 \n",
      "\n",
      "Completed near join in admin area 223. Thu Jan 26 10:06:20 2023 \n",
      "\n",
      "Completed near join in admin area 224. Thu Jan 26 10:06:20 2023 \n",
      "\n",
      "Completed near join in admin area 225. Thu Jan 26 10:06:21 2023 \n",
      "\n",
      "Completed near join in admin area 226. Thu Jan 26 10:06:21 2023 \n",
      "\n",
      "Completed near join in admin area 227. Thu Jan 26 10:06:21 2023 \n",
      "\n",
      "Completed near join in admin area 228. Thu Jan 26 10:06:21 2023 \n",
      "\n",
      "Completed near join in admin area 229. Thu Jan 26 10:06:21 2023 \n",
      "\n",
      "Completed near join in admin area 230. Thu Jan 26 10:06:21 2023 \n",
      "\n",
      "Completed near join in admin area 231. Thu Jan 26 10:06:21 2023 \n",
      "\n",
      "Completed near join in admin area 232. Thu Jan 26 10:06:22 2023 \n",
      "\n",
      "Completed near join in admin area 233. Thu Jan 26 10:06:22 2023 \n",
      "\n",
      "Completed near join in admin area 234. Thu Jan 26 10:06:22 2023 \n",
      "\n",
      "Completed near join in admin area 235. Thu Jan 26 10:06:22 2023 \n",
      "\n",
      "Completed near join in admin area 236. Thu Jan 26 10:06:22 2023 \n",
      "\n",
      "Completed near join in admin area 237. Thu Jan 26 10:06:22 2023 \n",
      "\n",
      "Completed near join in admin area 238. Thu Jan 26 10:06:23 2023 \n",
      "\n",
      "Completed near join in admin area 239. Thu Jan 26 10:06:23 2023 \n",
      "\n",
      "Completed near join in admin area 240. Thu Jan 26 10:06:23 2023 \n",
      "\n",
      "Completed near join in admin area 241. Thu Jan 26 10:06:23 2023 \n",
      "\n",
      "Completed near join in admin area 242. Thu Jan 26 10:06:23 2023 \n",
      "\n",
      "Completed near join in admin area 243. Thu Jan 26 10:06:23 2023 \n",
      "\n",
      "Completed near join in admin area 244. Thu Jan 26 10:06:24 2023 \n",
      "\n",
      "Completed near join in admin area 245. Thu Jan 26 10:06:24 2023 \n",
      "\n",
      "Completed near join in admin area 246. Thu Jan 26 10:06:24 2023 \n",
      "\n",
      "Completed near join in admin area 247. Thu Jan 26 10:06:24 2023 \n",
      "\n",
      "Completed near join in admin area 248. Thu Jan 26 10:06:24 2023 \n",
      "\n",
      "Completed near join in admin area 249. Thu Jan 26 10:06:24 2023 \n",
      "\n",
      "Completed near join in admin area 250. Thu Jan 26 10:06:25 2023 \n",
      "\n",
      "Completed near join in admin area 251. Thu Jan 26 10:06:25 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed near join in admin area 252. Thu Jan 26 10:06:25 2023 \n",
      "\n",
      "Completed near join in admin area 253. Thu Jan 26 10:06:25 2023 \n",
      "\n",
      "Completed near join in admin area 254. Thu Jan 26 10:06:26 2023 \n",
      "\n",
      "Completed near join in admin area 255. Thu Jan 26 10:06:27 2023 \n",
      "\n",
      "Completed near join in admin area 256. Thu Jan 26 10:06:27 2023 \n",
      "\n",
      "Completed near join in admin area 257. Thu Jan 26 10:06:27 2023 \n",
      "\n",
      "Completed near join in admin area 258. Thu Jan 26 10:06:27 2023 \n",
      "\n",
      "Completed near join in admin area 259. Thu Jan 26 10:06:27 2023 \n",
      "\n",
      "Completed near join in admin area 260. Thu Jan 26 10:06:27 2023 \n",
      "\n",
      "Completed near join in admin area 261. Thu Jan 26 10:06:28 2023 \n",
      "\n",
      "Completed near join in admin area 262. Thu Jan 26 10:06:28 2023 \n",
      "\n",
      "Completed near join in admin area 263. Thu Jan 26 10:06:28 2023 \n",
      "\n",
      "Completed near join in admin area 264. Thu Jan 26 10:06:28 2023 \n",
      "\n",
      "Completed near join in admin area 265. Thu Jan 26 10:06:28 2023 \n",
      "\n",
      "Completed near join in admin area 266. Thu Jan 26 10:06:28 2023 \n",
      "\n",
      "Completed near join in admin area 267. Thu Jan 26 10:06:28 2023 \n",
      "\n",
      "Completed near join in admin area 268. Thu Jan 26 10:06:29 2023 \n",
      "\n",
      "Completed near join in admin area 269. Thu Jan 26 10:06:29 2023 \n",
      "\n",
      "Completed near join in admin area 270. Thu Jan 26 10:06:29 2023 \n",
      "\n",
      "Completed near join in admin area 271. Thu Jan 26 10:06:29 2023 \n",
      "\n",
      "Completed near join in admin area 272. Thu Jan 26 10:06:29 2023 \n",
      "\n",
      "Completed near join in admin area 273. Thu Jan 26 10:06:30 2023 \n",
      "\n",
      "Completed near join in admin area 274. Thu Jan 26 10:06:30 2023 \n",
      "\n",
      "Completed near join in admin area 275. Thu Jan 26 10:06:30 2023 \n",
      "\n",
      "Completed near join in admin area 276. Thu Jan 26 10:06:30 2023 \n",
      "\n",
      "Completed near join in admin area 277. Thu Jan 26 10:06:30 2023 \n",
      "\n",
      "Completed near join in admin area 278. Thu Jan 26 10:06:30 2023 \n",
      "\n",
      "Completed near join in admin area 279. Thu Jan 26 10:06:30 2023 \n",
      "\n",
      "Completed near join in admin area 280. Thu Jan 26 10:06:31 2023 \n",
      "\n",
      "Completed near join in admin area 281. Thu Jan 26 10:06:31 2023 \n",
      "\n",
      "Completed near join in admin area 282. Thu Jan 26 10:06:31 2023 \n",
      "\n",
      "Completed near join in admin area 283. Thu Jan 26 10:06:31 2023 \n",
      "\n",
      "Completed near join in admin area 284. Thu Jan 26 10:06:32 2023 \n",
      "\n",
      "Completed near join in admin area 285. Thu Jan 26 10:06:32 2023 \n",
      "\n",
      "Completed near join in admin area 286. Thu Jan 26 10:06:32 2023 \n",
      "\n",
      "Completed near join in admin area 287. Thu Jan 26 10:06:32 2023 \n",
      "\n",
      "Completed near join in admin area 288. Thu Jan 26 10:06:32 2023 \n",
      "\n",
      "Completed near join in admin area 289. Thu Jan 26 10:06:32 2023 \n",
      "\n",
      "Completed near join in admin area 290. Thu Jan 26 10:06:32 2023 \n",
      "\n",
      "Completed near join in admin area 291. Thu Jan 26 10:06:33 2023 \n",
      "\n",
      "Completed near join in admin area 292. Thu Jan 26 10:06:33 2023 \n",
      "\n",
      "Completed near join in admin area 293. Thu Jan 26 10:06:33 2023 \n",
      "\n",
      "Completed near join in admin area 294. Thu Jan 26 10:06:33 2023 \n",
      "\n",
      "Completed near join in admin area 295. Thu Jan 26 10:06:33 2023 \n",
      "\n",
      "Completed near join in admin area 296. Thu Jan 26 10:06:34 2023 \n",
      "\n",
      "Completed near join in admin area 297. Thu Jan 26 10:06:34 2023 \n",
      "\n",
      "Completed near join in admin area 298. Thu Jan 26 10:06:34 2023 \n",
      "\n",
      "Completed near join in admin area 299. Thu Jan 26 10:06:34 2023 \n",
      "\n",
      "Completed near join in admin area 300. Thu Jan 26 10:06:34 2023 \n",
      "\n",
      "Completed near join in admin area 301. Thu Jan 26 10:06:35 2023 \n",
      "\n",
      "Completed near join in admin area 302. Thu Jan 26 10:06:35 2023 \n",
      "\n",
      "Completed near join in admin area 303. Thu Jan 26 10:06:35 2023 \n",
      "\n",
      "Completed near join in admin area 304. Thu Jan 26 10:06:35 2023 \n",
      "\n",
      "Completed near join in admin area 305. Thu Jan 26 10:06:35 2023 \n",
      "\n",
      "Completed near join in admin area 306. Thu Jan 26 10:06:36 2023 \n",
      "\n",
      "Completed near join in admin area 307. Thu Jan 26 10:06:36 2023 \n",
      "\n",
      "Completed near join in admin area 308. Thu Jan 26 10:06:36 2023 \n",
      "\n",
      "Completed near join in admin area 309. Thu Jan 26 10:06:36 2023 \n",
      "\n",
      "Completed near join in admin area 310. Thu Jan 26 10:06:36 2023 \n",
      "\n",
      "Completed near join in admin area 311. Thu Jan 26 10:06:36 2023 \n",
      "\n",
      "Completed near join in admin area 312. Thu Jan 26 10:06:36 2023 \n",
      "\n",
      "Completed near join in admin area 313. Thu Jan 26 10:06:37 2023 \n",
      "\n",
      "Completed near join in admin area 314. Thu Jan 26 10:06:37 2023 \n",
      "\n",
      "Completed near join in admin area 315. Thu Jan 26 10:06:37 2023 \n",
      "\n",
      "Completed near join in admin area 316. Thu Jan 26 10:06:37 2023 \n",
      "\n",
      "Completed near join in admin area 317. Thu Jan 26 10:06:37 2023 \n",
      "\n",
      "Completed near join in admin area 318. Thu Jan 26 10:06:38 2023 \n",
      "\n",
      "Completed near join in admin area 319. Thu Jan 26 10:06:38 2023 \n",
      "\n",
      "Completed near join in admin area 320. Thu Jan 26 10:06:38 2023 \n",
      "\n",
      "Completed near join in admin area 321. Thu Jan 26 10:06:38 2023 \n",
      "\n",
      "Completed near join in admin area 322. Thu Jan 26 10:06:38 2023 \n",
      "\n",
      "Completed near join in admin area 323. Thu Jan 26 10:06:38 2023 \n",
      "\n",
      "Completed near join in admin area 324. Thu Jan 26 10:06:39 2023 \n",
      "\n",
      "Completed near join in admin area 325. Thu Jan 26 10:06:39 2023 \n",
      "\n",
      "Completed near join in admin area 326. Thu Jan 26 10:06:39 2023 \n",
      "\n",
      "Completed near join in admin area 327. Thu Jan 26 10:06:39 2023 \n",
      "\n",
      "Completed near join in admin area 328. Thu Jan 26 10:06:39 2023 \n",
      "\n",
      "Completed near join in admin area 329. Thu Jan 26 10:06:39 2023 \n",
      "\n",
      "Completed near join in admin area 330. Thu Jan 26 10:06:40 2023 \n",
      "\n",
      "Completed near join in admin area 331. Thu Jan 26 10:06:40 2023 \n",
      "\n",
      "Completed near join in admin area 332. Thu Jan 26 10:06:40 2023 \n",
      "\n",
      "Completed near join in admin area 333. Thu Jan 26 10:06:40 2023 \n",
      "\n",
      "Completed near join in admin area 334. Thu Jan 26 10:06:40 2023 \n",
      "\n",
      "Completed near join in admin area 335. Thu Jan 26 10:06:40 2023 \n",
      "\n",
      "Completed near join in admin area 336. Thu Jan 26 10:06:41 2023 \n",
      "\n",
      "Completed near join in admin area 337. Thu Jan 26 10:06:41 2023 \n",
      "\n",
      "Completed near join in admin area 338. Thu Jan 26 10:06:41 2023 \n",
      "\n",
      "Completed near join in admin area 339. Thu Jan 26 10:06:41 2023 \n",
      "\n",
      "Completed near join in admin area 340. Thu Jan 26 10:06:41 2023 \n",
      "\n",
      "Completed near join in admin area 341. Thu Jan 26 10:06:41 2023 \n",
      "\n",
      "Completed near join in admin area 342. Thu Jan 26 10:06:42 2023 \n",
      "\n",
      "Completed near join in admin area 343. Thu Jan 26 10:06:42 2023 \n",
      "\n",
      "Completed near join in admin area 344. Thu Jan 26 10:06:42 2023 \n",
      "\n",
      "Completed near join in admin area 345. Thu Jan 26 10:06:42 2023 \n",
      "\n",
      "Completed near join in admin area 346. Thu Jan 26 10:06:42 2023 \n",
      "\n",
      "Completed near join in admin area 347. Thu Jan 26 10:06:42 2023 \n",
      "\n",
      "Completed near join in admin area 348. Thu Jan 26 10:06:42 2023 \n",
      "\n",
      "Completed near join in admin area 349. Thu Jan 26 10:06:43 2023 \n",
      "\n",
      "Completed near join in admin area 350. Thu Jan 26 10:06:43 2023 \n",
      "\n",
      "Completed near join in admin area 351. Thu Jan 26 10:06:43 2023 \n",
      "\n",
      "Completed near join in admin area 647. Thu Jan 26 10:28:31 2023 \n",
      "\n",
      "Completed near join for all ADMs. Thu Jan 26 10:28:31 2023 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for ID in ADM_IDs:\n",
    "    WSFE_shard = WSFE_ADM.loc[WSFE_ADM['ADM_ID'] == ID]\n",
    "    GRID3_shard = GRID3_ADM.loc[GRID3_ADM['ADM_ID'] == ID]\n",
    "    WSFE_GRID3_shard = gpd.sjoin_nearest(WSFE_shard, \n",
    "                                         GRID3_shard, \n",
    "                                         how='inner',\n",
    "                                         max_distance=500)\n",
    "    Bounded = pd.concat([Bounded, WSFE_GRID3_shard])\n",
    "    print('Completed near join in admin area %s. %s \\n' % (ID, time.ctime()))\n",
    "print('Completed near join for all ADMs. %s \\n' % time.ctime())\n",
    "\n",
    "del WSFE_shard, GRID3_shard, WSFE_GRID3_shard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "631c4985",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>Bounded_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>geometry</th>\n",
       "      <th>G3_Area</th>\n",
       "      <th>year</th>\n",
       "      <th>WSFE_ID</th>\n",
       "      <th>ADM_ID_left</th>\n",
       "      <th>index_right</th>\n",
       "      <th>ADM_ID_right</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>838100</th>\n",
       "      <td>36811</td>\n",
       "      <td>37034</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-3117713.916 1272802.609, -3117684.7...</td>\n",
       "      <td>7.998835</td>\n",
       "      <td>2008</td>\n",
       "      <td>838100.0</td>\n",
       "      <td>216.0</td>\n",
       "      <td>37034.0</td>\n",
       "      <td>216.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239362</th>\n",
       "      <td>533209</td>\n",
       "      <td>538156</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2602301.746 1456724.293, -2602272.5...</td>\n",
       "      <td>0.100283</td>\n",
       "      <td>2010</td>\n",
       "      <td>239362.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>538156.0</td>\n",
       "      <td>31.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>549551</th>\n",
       "      <td>130003</td>\n",
       "      <td>130871</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2932713.597 1393347.254, -2932684.4...</td>\n",
       "      <td>1.051271</td>\n",
       "      <td>2009</td>\n",
       "      <td>549551.0</td>\n",
       "      <td>228.0</td>\n",
       "      <td>130871.0</td>\n",
       "      <td>228.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70095</th>\n",
       "      <td>457760</td>\n",
       "      <td>462410</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2850096.058 1560448.077, -2850066.9...</td>\n",
       "      <td>0.292350</td>\n",
       "      <td>2000</td>\n",
       "      <td>70095.0</td>\n",
       "      <td>225.0</td>\n",
       "      <td>462410.0</td>\n",
       "      <td>225.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>354093</th>\n",
       "      <td>227047</td>\n",
       "      <td>229233</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2773746.258 1436113.637, -2773717.1...</td>\n",
       "      <td>378.496507</td>\n",
       "      <td>1991</td>\n",
       "      <td>354093.0</td>\n",
       "      <td>218.0</td>\n",
       "      <td>229233.0</td>\n",
       "      <td>218.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>186503</th>\n",
       "      <td>388701</td>\n",
       "      <td>392517</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2987286.651 1488966.706, -2987257.4...</td>\n",
       "      <td>1.047871</td>\n",
       "      <td>1985</td>\n",
       "      <td>186503.0</td>\n",
       "      <td>215.0</td>\n",
       "      <td>392517.0</td>\n",
       "      <td>215.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>580041</th>\n",
       "      <td>247671</td>\n",
       "      <td>250117</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2686406.051 1379674.838, -2686376.8...</td>\n",
       "      <td>0.222662</td>\n",
       "      <td>2007</td>\n",
       "      <td>580041.0</td>\n",
       "      <td>148.0</td>\n",
       "      <td>250117.0</td>\n",
       "      <td>148.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>580723</th>\n",
       "      <td>117196</td>\n",
       "      <td>118008</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-3058884.631 1379383.315, -3058797.1...</td>\n",
       "      <td>7.990336</td>\n",
       "      <td>1985</td>\n",
       "      <td>580723.0</td>\n",
       "      <td>159.0</td>\n",
       "      <td>118008.0</td>\n",
       "      <td>159.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>708101</th>\n",
       "      <td>51186</td>\n",
       "      <td>51489</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-3026175.782 1326530.246, -3026146.6...</td>\n",
       "      <td>1.626622</td>\n",
       "      <td>2000</td>\n",
       "      <td>708101.0</td>\n",
       "      <td>269.0</td>\n",
       "      <td>51489.0</td>\n",
       "      <td>269.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>609055</th>\n",
       "      <td>247085</td>\n",
       "      <td>249500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2645942.697 1371220.679, -2645913.5...</td>\n",
       "      <td>29.359090</td>\n",
       "      <td>2009</td>\n",
       "      <td>609055.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>249500.0</td>\n",
       "      <td>296.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>617512</th>\n",
       "      <td>247078</td>\n",
       "      <td>249491</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2685618.939 1368684.432, -2685589.7...</td>\n",
       "      <td>5.888645</td>\n",
       "      <td>2000</td>\n",
       "      <td>617512.0</td>\n",
       "      <td>211.0</td>\n",
       "      <td>249491.0</td>\n",
       "      <td>211.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>572094</th>\n",
       "      <td>37255</td>\n",
       "      <td>37490</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-3108006.209 1383289.720, -3107977.0...</td>\n",
       "      <td>0.407081</td>\n",
       "      <td>1995</td>\n",
       "      <td>572094.0</td>\n",
       "      <td>206.0</td>\n",
       "      <td>37490.0</td>\n",
       "      <td>206.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>582883</th>\n",
       "      <td>129962</td>\n",
       "      <td>130827</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2968366.826 1378596.204, -2968308.5...</td>\n",
       "      <td>1.045322</td>\n",
       "      <td>2005</td>\n",
       "      <td>582883.0</td>\n",
       "      <td>319.0</td>\n",
       "      <td>130827.0</td>\n",
       "      <td>319.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>800474</th>\n",
       "      <td>351201</td>\n",
       "      <td>354806</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2528517.345 1292392.936, -2528488.1...</td>\n",
       "      <td>4.834825</td>\n",
       "      <td>2008</td>\n",
       "      <td>800474.0</td>\n",
       "      <td>146.0</td>\n",
       "      <td>354806.0</td>\n",
       "      <td>146.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>583411</th>\n",
       "      <td>188703</td>\n",
       "      <td>190117</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2817766.189 1378392.138, -2817707.8...</td>\n",
       "      <td>0.112181</td>\n",
       "      <td>2015</td>\n",
       "      <td>583411.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>190117.0</td>\n",
       "      <td>46.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>697172</th>\n",
       "      <td>265723</td>\n",
       "      <td>268424</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2684598.610 1331573.589, -2684540.3...</td>\n",
       "      <td>0.823509</td>\n",
       "      <td>2005</td>\n",
       "      <td>697172.0</td>\n",
       "      <td>105.0</td>\n",
       "      <td>268424.0</td>\n",
       "      <td>105.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>438969</th>\n",
       "      <td>138001</td>\n",
       "      <td>138973</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2978570.121 1425152.383, -2978540.9...</td>\n",
       "      <td>0.142776</td>\n",
       "      <td>1985</td>\n",
       "      <td>438969.0</td>\n",
       "      <td>64.0</td>\n",
       "      <td>138973.0</td>\n",
       "      <td>64.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>422453</th>\n",
       "      <td>310052</td>\n",
       "      <td>313257</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2639587.502 1427776.087, -2639558.3...</td>\n",
       "      <td>0.169971</td>\n",
       "      <td>2009</td>\n",
       "      <td>422453.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>313257.0</td>\n",
       "      <td>163.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109028</th>\n",
       "      <td>453764</td>\n",
       "      <td>458345</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2854439.747 1533802.901, -2854410.5...</td>\n",
       "      <td>13.404762</td>\n",
       "      <td>2015</td>\n",
       "      <td>109028.0</td>\n",
       "      <td>111.0</td>\n",
       "      <td>458345.0</td>\n",
       "      <td>111.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>471562</th>\n",
       "      <td>309964</td>\n",
       "      <td>313161</td>\n",
       "      <td>NaN</td>\n",
       "      <td>POLYGON ((-2639558.349 1419759.212, -2639529.1...</td>\n",
       "      <td>1.491495</td>\n",
       "      <td>2013</td>\n",
       "      <td>471562.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>313161.0</td>\n",
       "      <td>163.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Sett_ID  Bounded_ID  ADM_ID  \\\n",
       "838100    36811       37034     NaN   \n",
       "239362   533209      538156     NaN   \n",
       "549551   130003      130871     NaN   \n",
       "70095    457760      462410     NaN   \n",
       "354093   227047      229233     NaN   \n",
       "186503   388701      392517     NaN   \n",
       "580041   247671      250117     NaN   \n",
       "580723   117196      118008     NaN   \n",
       "708101    51186       51489     NaN   \n",
       "609055   247085      249500     NaN   \n",
       "617512   247078      249491     NaN   \n",
       "572094    37255       37490     NaN   \n",
       "582883   129962      130827     NaN   \n",
       "800474   351201      354806     NaN   \n",
       "583411   188703      190117     NaN   \n",
       "697172   265723      268424     NaN   \n",
       "438969   138001      138973     NaN   \n",
       "422453   310052      313257     NaN   \n",
       "109028   453764      458345     NaN   \n",
       "471562   309964      313161     NaN   \n",
       "\n",
       "                                                 geometry     G3_Area  year  \\\n",
       "838100  POLYGON ((-3117713.916 1272802.609, -3117684.7...    7.998835  2008   \n",
       "239362  POLYGON ((-2602301.746 1456724.293, -2602272.5...    0.100283  2010   \n",
       "549551  POLYGON ((-2932713.597 1393347.254, -2932684.4...    1.051271  2009   \n",
       "70095   POLYGON ((-2850096.058 1560448.077, -2850066.9...    0.292350  2000   \n",
       "354093  POLYGON ((-2773746.258 1436113.637, -2773717.1...  378.496507  1991   \n",
       "186503  POLYGON ((-2987286.651 1488966.706, -2987257.4...    1.047871  1985   \n",
       "580041  POLYGON ((-2686406.051 1379674.838, -2686376.8...    0.222662  2007   \n",
       "580723  POLYGON ((-3058884.631 1379383.315, -3058797.1...    7.990336  1985   \n",
       "708101  POLYGON ((-3026175.782 1326530.246, -3026146.6...    1.626622  2000   \n",
       "609055  POLYGON ((-2645942.697 1371220.679, -2645913.5...   29.359090  2009   \n",
       "617512  POLYGON ((-2685618.939 1368684.432, -2685589.7...    5.888645  2000   \n",
       "572094  POLYGON ((-3108006.209 1383289.720, -3107977.0...    0.407081  1995   \n",
       "582883  POLYGON ((-2968366.826 1378596.204, -2968308.5...    1.045322  2005   \n",
       "800474  POLYGON ((-2528517.345 1292392.936, -2528488.1...    4.834825  2008   \n",
       "583411  POLYGON ((-2817766.189 1378392.138, -2817707.8...    0.112181  2015   \n",
       "697172  POLYGON ((-2684598.610 1331573.589, -2684540.3...    0.823509  2005   \n",
       "438969  POLYGON ((-2978570.121 1425152.383, -2978540.9...    0.142776  1985   \n",
       "422453  POLYGON ((-2639587.502 1427776.087, -2639558.3...    0.169971  2009   \n",
       "109028  POLYGON ((-2854439.747 1533802.901, -2854410.5...   13.404762  2015   \n",
       "471562  POLYGON ((-2639558.349 1419759.212, -2639529.1...    1.491495  2013   \n",
       "\n",
       "         WSFE_ID  ADM_ID_left  index_right  ADM_ID_right  \n",
       "838100  838100.0        216.0      37034.0         216.0  \n",
       "239362  239362.0         31.0     538156.0          31.0  \n",
       "549551  549551.0        228.0     130871.0         228.0  \n",
       "70095    70095.0        225.0     462410.0         225.0  \n",
       "354093  354093.0        218.0     229233.0         218.0  \n",
       "186503  186503.0        215.0     392517.0         215.0  \n",
       "580041  580041.0        148.0     250117.0         148.0  \n",
       "580723  580723.0        159.0     118008.0         159.0  \n",
       "708101  708101.0        269.0      51489.0         269.0  \n",
       "609055  609055.0        296.0     249500.0         296.0  \n",
       "617512  617512.0        211.0     249491.0         211.0  \n",
       "572094  572094.0        206.0      37490.0         206.0  \n",
       "582883  582883.0        319.0     130827.0         319.0  \n",
       "800474  800474.0        146.0     354806.0         146.0  \n",
       "583411  583411.0         46.0     190117.0          46.0  \n",
       "697172  697172.0        105.0     268424.0         105.0  \n",
       "438969  438969.0         64.0     138973.0          64.0  \n",
       "422453  422453.0        163.0     313257.0         163.0  \n",
       "109028  109028.0        111.0     458345.0         111.0  \n",
       "471562  471562.0        163.0     313161.0         163.0  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Bounded.sample(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c5b56278",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 921485 entries, 211201 to 922082\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count   Dtype   \n",
      "---  ------        --------------   -----   \n",
      " 0   Sett_ID       921485 non-null  int64   \n",
      " 1   Bounded_ID    921485 non-null  int64   \n",
      " 2   ADM_ID        0 non-null       float64 \n",
      " 3   geometry      921485 non-null  geometry\n",
      " 4   G3_Area       921485 non-null  float64 \n",
      " 5   year          921485 non-null  int32   \n",
      " 6   WSFE_ID       921485 non-null  float64 \n",
      " 7   ADM_ID_left   921485 non-null  float64 \n",
      " 8   index_right   921485 non-null  float64 \n",
      " 9   ADM_ID_right  921485 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int32(1), int64(2)\n",
      "memory usage: 73.8 MB\n"
     ]
    }
   ],
   "source": [
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "fdc68243",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 921485 entries, 211201 to 922082\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count   Dtype   \n",
      "---  ------        --------------   -----   \n",
      " 0   Sett_ID       921485 non-null  int64   \n",
      " 1   Bounded_ID    921485 non-null  int64   \n",
      " 2   ADM_ID        0 non-null       float64 \n",
      " 3   geometry      921485 non-null  geometry\n",
      " 4   G3_Area       921485 non-null  float64 \n",
      " 5   year          921485 non-null  int32   \n",
      " 6   WSFE_ID       921485 non-null  float64 \n",
      " 7   ADM_ID_left   921485 non-null  float64 \n",
      " 8   index_right   921485 non-null  float64 \n",
      " 9   ADM_ID_right  921485 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int32(1), int64(2)\n",
      "memory usage: 73.8 MB\n"
     ]
    }
   ],
   "source": [
    "# Remove WSFE features that did not match any GRID3 settlements.\n",
    "Bounded = Bounded.loc[~Bounded['Sett_ID'].isna()]\n",
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c5736e5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "del GRID3_ADM, ADM_IDs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "059bd29d",
   "metadata": {},
   "source": [
    "### 4.2 Remove duplicates: where buildup polygons intersected with more than one GRID3 settlement extent.\n",
    "This happens when the first dataset (WSFE) intersects (distance = 0) with more than one feature of the second dataset (GRID3). More common for large cities. For example, Yaoundé, CMN has a large contiguous 1985 WSFE polygon which overlaps several small GRID3 features that are not Yaoundé."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "49786b23",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 1731\n"
     ]
    }
   ],
   "source": [
    "# The first number should always be zero. \n",
    "# The second tells us whether/how many WSFE polygons were duplicated by the Near join.\n",
    "\n",
    "print(len(WSFE_ADM[WSFE_ADM.duplicated('WSFE_ID')]), len(Bounded[Bounded.duplicated('WSFE_ID')]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0b92f155",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "Int64Index: 919754 entries, 922082 to 862313\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count   Dtype   \n",
      "---  ------        --------------   -----   \n",
      " 0   Sett_ID       919754 non-null  int64   \n",
      " 1   Bounded_ID    919754 non-null  int64   \n",
      " 2   ADM_ID        0 non-null       float64 \n",
      " 3   geometry      919754 non-null  geometry\n",
      " 4   G3_Area       919754 non-null  float64 \n",
      " 5   year          919754 non-null  int32   \n",
      " 6   WSFE_ID       919754 non-null  float64 \n",
      " 7   ADM_ID_left   919754 non-null  float64 \n",
      " 8   index_right   919754 non-null  float64 \n",
      " 9   ADM_ID_right  919754 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int32(1), int64(2)\n",
      "memory usage: 73.7 MB\n"
     ]
    }
   ],
   "source": [
    "# If there are duplicate WSFE_IDs, then we need to choose between them.\n",
    "# We'll pick the one that joined with the largest GRID3 polygon.\n",
    "# To do that, we can just sort the dataframe by GRID3 areas, then drop_duplicates. \n",
    "# It will retain the first row of each WSFE_ID group.\n",
    "Bounded = Bounded.sort_values('G3_Area', ascending=False).drop_duplicates(['WSFE_ID'])\n",
    "Bounded.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "cd6141b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "print(len(Bounded[Bounded.duplicated('WSFE_ID')]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "7d22854d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 180796 entries, 0 to 180795\n",
      "Data columns (total 10 columns):\n",
      " #   Column        Non-Null Count   Dtype   \n",
      "---  ------        --------------   -----   \n",
      " 0   year          180796 non-null  int64   \n",
      " 1   Bounded_ID    180796 non-null  int64   \n",
      " 2   geometry      180796 non-null  geometry\n",
      " 3   Sett_ID       180796 non-null  int64   \n",
      " 4   ADM_ID        0 non-null       float64 \n",
      " 5   G3_Area       180796 non-null  float64 \n",
      " 6   WSFE_ID       180796 non-null  float64 \n",
      " 7   ADM_ID_left   180796 non-null  float64 \n",
      " 8   index_right   180796 non-null  float64 \n",
      " 9   ADM_ID_right  180796 non-null  float64 \n",
      "dtypes: float64(6), geometry(1), int64(3)\n",
      "memory usage: 13.8 MB\n",
      "None         year  Bounded_ID                                           geometry  \\\n",
      "75702   2003      138845  POLYGON ((-2994079.130 1410430.485, -2994049.9...   \n",
      "95142   2006      249989  MULTIPOLYGON (((-2681362.708 1353087.966, -268...   \n",
      "2677    1985      138901  MULTIPOLYGON (((-2922714.368 1399585.840, -292...   \n",
      "82141   2004      181853  MULTIPOLYGON (((-2879598.158 1469288.922, -287...   \n",
      "116706  2009      392561  POLYGON ((-3030869.297 1501968.619, -3030840.1...   \n",
      "175069  2015      317550  MULTIPOLYGON (((-2622912.402 1420138.192, -262...   \n",
      "150868  2013      322233  POLYGON ((-2611309.798 1417135.508, -2611280.6...   \n",
      "7546    1985      319023  MULTIPOLYGON (((-2649353.513 1431361.817, -264...   \n",
      "112468  2009       37082  MULTIPOLYGON (((-3109930.259 1272190.411, -310...   \n",
      "41143   1996      419629  MULTIPOLYGON (((-2721680.300 1518993.546, -272...   \n",
      "\n",
      "        Sett_ID  ADM_ID    G3_Area   WSFE_ID  ADM_ID_left  index_right  \\\n",
      "75702    137875     NaN   0.358639  500356.0        221.0     138845.0   \n",
      "95142    247551     NaN   1.178749  653447.0        346.0     249989.0   \n",
      "2677     137930     NaN   1.316425  535838.0        294.0     138901.0   \n",
      "82141    180521     NaN   0.537108  217488.0        102.0     181853.0   \n",
      "116706   388745     NaN   0.404531  168469.0         83.0     392561.0   \n",
      "175069   314268     NaN   0.061190  470789.0         22.0     317550.0   \n",
      "150868   318859     NaN   0.271954  480479.0        300.0     322233.0   \n",
      "7546     315706     NaN   0.084985  390799.0          2.0     319023.0   \n",
      "112468    36859     NaN   0.329744  839867.0        164.0      37082.0   \n",
      "41143    415487     NaN  27.276095  133728.0        131.0     419629.0   \n",
      "\n",
      "        ADM_ID_right  \n",
      "75702          221.0  \n",
      "95142          346.0  \n",
      "2677           294.0  \n",
      "82141          102.0  \n",
      "116706          83.0  \n",
      "175069          22.0  \n",
      "150868         300.0  \n",
      "7546             2.0  \n",
      "112468         164.0  \n",
      "41143          131.0  \n"
     ]
    }
   ],
   "source": [
    "# Now we can dissolve with the WSFE years, now that we can group them by their administratively split ID.\n",
    "Bounded = Bounded.dissolve(by=['year', 'Bounded_ID'], as_index=False)\n",
    "print(Bounded.info(), Bounded.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e45ca6e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        ADM_ID  year  Bounded_ID  Sett_ID  \\\n",
      "131339      30  2011      167131   165921   \n",
      "136676     272  2011      586607   581299   \n",
      "73014      219  2002      461528   456905   \n",
      "20446      329  1990      157793   156712   \n",
      "38087       19  1995      463087   458422   \n",
      "171714     151  2015      163923   162770   \n",
      "14519       97  1988      250058   247615   \n",
      "163285     324  2014      312964   309772   \n",
      "163266      67  2014      312844   309661   \n",
      "54262      320  1999      413412   409365   \n",
      "\n",
      "                                                 geometry  \n",
      "131339  POLYGON ((-2811002.861 1305453.153, -2810973.7...  \n",
      "136676  POLYGON ((-2568893.242 1623737.660, -2568864.0...  \n",
      "73014   MULTIPOLYGON (((-2863302.038 1572283.900, -286...  \n",
      "20446   POLYGON ((-2925629.595 1255340.398, -2925600.4...  \n",
      "38087   MULTIPOLYGON (((-2848638.445 1595459.956, -284...  \n",
      "171714  POLYGON ((-2907526.034 1292043.108, -2907467.7...  \n",
      "14519   MULTIPOLYGON (((-2671276.021 1370579.329, -267...  \n",
      "163285  MULTIPOLYGON (((-2636526.513 1402530.219, -263...  \n",
      "163266  MULTIPOLYGON (((-2650811.127 1393288.949, -265...  \n",
      "54262   MULTIPOLYGON (((-2841583.595 1503163.863, -284...  \n"
     ]
    }
   ],
   "source": [
    "# Clean up and save to file.\n",
    "Bounded = Bounded[['ADM_ID_left', 'year', 'Bounded_ID', 'Sett_ID', 'geometry']].rename(columns={\"ADM_ID_left\": \"ADM_ID\"})\n",
    "Bounded = Bounded.astype({\"ADM_ID\":'int', \"Bounded_ID\":'int', \"Sett_ID\":'int', \"year\":'int'})\n",
    "print(Bounded.sample(10))\n",
    "Bounded.to_file(\n",
    "    driver='GPKG', filename=r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Bounded')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "75e43acf",
   "metadata": {},
   "outputs": [],
   "source": [
    "del WSFE_ADM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea6eadab",
   "metadata": {},
   "source": [
    "### 4.3 BOUNDLESS SETTLEMENTS: Dissolve features that were split by an ADM boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "f2638d9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 179618 entries, 0 to 179617\n",
      "Data columns (total 5 columns):\n",
      " #   Column      Non-Null Count   Dtype   \n",
      "---  ------      --------------   -----   \n",
      " 0   year        179618 non-null  int64   \n",
      " 1   Sett_ID     179618 non-null  int64   \n",
      " 2   geometry    179618 non-null  geometry\n",
      " 3   ADM_ID      179618 non-null  int32   \n",
      " 4   Bounded_ID  179618 non-null  int32   \n",
      "dtypes: geometry(1), int32(2), int64(2)\n",
      "memory usage: 5.5 MB\n",
      "None         year  Sett_ID                                           geometry  \\\n",
      "308     1985      580  MULTIPOLYGON (((-2997344.185 1204732.054, -299...   \n",
      "177785  2015   468152  MULTIPOLYGON (((-2825287.475 1597704.681, -282...   \n",
      "90270   2005   457940  POLYGON ((-2863855.931 1567123.948, -2863826.7...   \n",
      "130656  2011   180320  MULTIPOLYGON (((-2865342.697 1408273.217, -286...   \n",
      "71275   2002   314016  POLYGON ((-2633698.743 1418097.533, -2633669.5...   \n",
      "123679  2010   294894  POLYGON ((-2670547.215 1447249.805, -2670518.0...   \n",
      "70088   2002   166100  MULTIPOLYGON (((-2871872.806 1361338.059, -287...   \n",
      "30365   1994   203749  POLYGON ((-2853536.026 1428563.198, -2853506.8...   \n",
      "65588   2001   247900  MULTIPOLYGON (((-2626235.761 1365798.357, -262...   \n",
      "134412  2011   457593  POLYGON ((-2827940.332 1553743.055, -2827911.1...   \n",
      "\n",
      "        ADM_ID  Bounded_ID  \n",
      "308        226         586  \n",
      "177785     306      472900  \n",
      "90270      219      462593  \n",
      "130656     295      181642  \n",
      "71275      163      317297  \n",
      "123679     266      297838  \n",
      "70088      276      167313  \n",
      "30365      157      205306  \n",
      "65588      296      250350  \n",
      "134412     271      462236  \n"
     ]
    }
   ],
   "source": [
    "# Fragments of any bounded settlement will be combined into a single \"boundless\" settlement in this version.\n",
    "# It is based on their \"Sett_ID\", which is a direct loan from the GRID3 settlement features.\n",
    "Boundless = Bounded.dissolve(by=['year', 'Sett_ID'], as_index=False)\n",
    "print(Boundless.info(), Boundless.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "b76e2a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Clean up and save to file.\n",
    "Boundless.to_file(driver='GPKG', filename=r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Boundless')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbba12eb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c027eba",
   "metadata": {},
   "source": [
    "## 5. CUMULATIVE ANNUALIZED SETTLEMENT EXTENTS\n",
    "DISSOLVE BY YEAR SETS: Create separate feature layers of each cumulative year."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d70322",
   "metadata": {},
   "source": [
    "### 5.1 Define study years for each for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cda3995e",
   "metadata": {},
   "outputs": [],
   "source": [
    "Boundless = gpd.read_file(r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Boundless')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "60c3aff1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " [2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999]\n"
     ]
    }
   ],
   "source": [
    "ReversedStudyYears_to2014 = []\n",
    "for i in AllStudyYears:\n",
    "    ReversedStudyYears_to2014.insert(0,i)\n",
    "ReversedStudyYears_to2014.remove(2015)\n",
    "print('\\n\\n', ReversedStudyYears_to2014)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15de5331",
   "metadata": {},
   "source": [
    "### 5.2 Starting with main Boundless dataset, create a cumulative area feature layer for each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "b3c1be53",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subsetting to cumulative area for year: 1999. Thu Jan 26 11:10:46 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:10:46 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:11:14 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2000. Thu Jan 26 11:11:22 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:11:22 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:11:54 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2001. Thu Jan 26 11:12:03 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:12:03 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:12:37 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2002. Thu Jan 26 11:12:46 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:12:46 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:13:20 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2003. Thu Jan 26 11:13:29 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:13:29 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:14:05 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2004. Thu Jan 26 11:14:13 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:14:13 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:14:52 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2005. Thu Jan 26 11:15:01 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:15:01 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:15:39 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2006. Thu Jan 26 11:15:48 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:15:48 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:16:30 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2007. Thu Jan 26 11:16:39 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:16:39 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:17:21 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2008. Thu Jan 26 11:17:30 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:17:30 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:18:15 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2009. Thu Jan 26 11:18:24 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:18:24 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:19:09 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2010. Thu Jan 26 11:19:18 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:19:18 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:20:06 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2011. Thu Jan 26 11:20:16 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:20:16 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:21:06 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2012. Thu Jan 26 11:21:15 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:21:15 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:22:06 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2013. Thu Jan 26 11:22:15 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:22:15 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:23:09 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2014. Thu Jan 26 11:23:19 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:23:19 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:24:14 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2015. Thu Jan 26 11:24:24 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. Thu Jan 26 11:24:24 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\3467227044.py:6: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Boundless[Boundless['year'].between(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:25:22 2023\n",
      "\n",
      "Done with all years in set. Thu Jan 26 11:25:32 2023\n"
     ]
    }
   ],
   "source": [
    "# For each year in the growth stats study, we are taking features from all years prior to and including that year, \n",
    "# dissolving those features, and exporting as its own file.\n",
    "\n",
    "for item in AllStudyYears:\n",
    "    print('Subsetting to cumulative area for year: %s. %s\\n' % (item, time.ctime()))\n",
    "    YearSet = Boundless[Boundless['year'].between(\n",
    "        1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n",
    "    print('Dissolving so that each unique settlement (Sett_ID) has a single cumulative WSFE feature. %s\\n' % time.ctime())\n",
    "    YearDissolve = YearSet.dissolve(by='Sett_ID', \n",
    "                                        aggfunc={\"year\": \"max\", \"ADM_ID\":\"min\"}, # Though ADM_ID should be matching every time.\n",
    "                                        as_index=False)\n",
    "    print('Write to file. %s\\n' % time.ctime())\n",
    "    YearName = ''.join(['Cu', str(item), '_Boundless'])\n",
    "    YearDissolve.to_file(driver='GPKG', filename=r'Results/CumulativeSettlements.gpkg', layer=YearName)\n",
    "    del YearSet, YearDissolve\n",
    "print(\"Done with all years in set. %s\" % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "307c4746",
   "metadata": {},
   "source": [
    "##### Join area information from each cumulative layer onto the latest year dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "b33dc9c9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cumulative layer for year 2014. Mon Feb 20 09:42:22 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:23 2023\n",
      "\n",
      "Merging variables from 2014 onto our latest year (2015) via table join. Mon Feb 20 09:42:23 2023\n",
      "\n",
      "Loading cumulative layer for year 2013. Mon Feb 20 09:42:23 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:25 2023\n",
      "\n",
      "Merging variables from 2013 onto our latest year (2015) via table join. Mon Feb 20 09:42:25 2023\n",
      "\n",
      "Loading cumulative layer for year 2012. Mon Feb 20 09:42:25 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:27 2023\n",
      "\n",
      "Merging variables from 2012 onto our latest year (2015) via table join. Mon Feb 20 09:42:28 2023\n",
      "\n",
      "Loading cumulative layer for year 2011. Mon Feb 20 09:42:28 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:29 2023\n",
      "\n",
      "Merging variables from 2011 onto our latest year (2015) via table join. Mon Feb 20 09:42:29 2023\n",
      "\n",
      "Loading cumulative layer for year 2010. Mon Feb 20 09:42:29 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:31 2023\n",
      "\n",
      "Merging variables from 2010 onto our latest year (2015) via table join. Mon Feb 20 09:42:31 2023\n",
      "\n",
      "Loading cumulative layer for year 2009. Mon Feb 20 09:42:31 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:33 2023\n",
      "\n",
      "Merging variables from 2009 onto our latest year (2015) via table join. Mon Feb 20 09:42:33 2023\n",
      "\n",
      "Loading cumulative layer for year 2008. Mon Feb 20 09:42:33 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:35 2023\n",
      "\n",
      "Merging variables from 2008 onto our latest year (2015) via table join. Mon Feb 20 09:42:35 2023\n",
      "\n",
      "Loading cumulative layer for year 2007. Mon Feb 20 09:42:35 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:37 2023\n",
      "\n",
      "Merging variables from 2007 onto our latest year (2015) via table join. Mon Feb 20 09:42:37 2023\n",
      "\n",
      "Loading cumulative layer for year 2006. Mon Feb 20 09:42:37 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:39 2023\n",
      "\n",
      "Merging variables from 2006 onto our latest year (2015) via table join. Mon Feb 20 09:42:39 2023\n",
      "\n",
      "Loading cumulative layer for year 2005. Mon Feb 20 09:42:39 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:41 2023\n",
      "\n",
      "Merging variables from 2005 onto our latest year (2015) via table join. Mon Feb 20 09:42:41 2023\n",
      "\n",
      "Loading cumulative layer for year 2004. Mon Feb 20 09:42:41 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:43 2023\n",
      "\n",
      "Merging variables from 2004 onto our latest year (2015) via table join. Mon Feb 20 09:42:43 2023\n",
      "\n",
      "Loading cumulative layer for year 2003. Mon Feb 20 09:42:43 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:44 2023\n",
      "\n",
      "Merging variables from 2003 onto our latest year (2015) via table join. Mon Feb 20 09:42:44 2023\n",
      "\n",
      "Loading cumulative layer for year 2002. Mon Feb 20 09:42:44 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:46 2023\n",
      "\n",
      "Merging variables from 2002 onto our latest year (2015) via table join. Mon Feb 20 09:42:46 2023\n",
      "\n",
      "Loading cumulative layer for year 2001. Mon Feb 20 09:42:46 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:48 2023\n",
      "\n",
      "Merging variables from 2001 onto our latest year (2015) via table join. Mon Feb 20 09:42:48 2023\n",
      "\n",
      "Loading cumulative layer for year 2000. Mon Feb 20 09:42:48 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:50 2023\n",
      "\n",
      "Merging variables from 2000 onto our latest year (2015) via table join. Mon Feb 20 09:42:50 2023\n",
      "\n",
      "Loading cumulative layer for year 1999. Mon Feb 20 09:42:50 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:51 2023\n",
      "\n",
      "Merging variables from 1999 onto our latest year (2015) via table join. Mon Feb 20 09:42:51 2023\n",
      "\n",
      "Done merging annualized areas onto latest year geometries. Saving to file. Mon Feb 20 09:42:51 2023\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23490 entries, 0 to 23489\n",
      "Data columns (total 20 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Sett_ID   23490 non-null  int64  \n",
      " 1   year      23490 non-null  int64  \n",
      " 2   ADM_ID    23490 non-null  int64  \n",
      " 3   AREA2015  23490 non-null  float64\n",
      " 4   AREA2014  22886 non-null  float64\n",
      " 5   AREA2013  22457 non-null  float64\n",
      " 6   AREA2012  21629 non-null  float64\n",
      " 7   AREA2011  21316 non-null  float64\n",
      " 8   AREA2010  20948 non-null  float64\n",
      " 9   AREA2009  20471 non-null  float64\n",
      " 10  AREA2008  20062 non-null  float64\n",
      " 11  AREA2007  19712 non-null  float64\n",
      " 12  AREA2006  19369 non-null  float64\n",
      " 13  AREA2005  19010 non-null  float64\n",
      " 14  AREA2004  18683 non-null  float64\n",
      " 15  AREA2003  18259 non-null  float64\n",
      " 16  AREA2002  17884 non-null  float64\n",
      " 17  AREA2001  17592 non-null  float64\n",
      " 18  AREA2000  17134 non-null  float64\n",
      " 19  AREA1999  16385 non-null  float64\n",
      "dtypes: float64(17), int64(3)\n",
      "memory usage: 3.8 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# The latest year in the study contains all settlements. Merge all other years' areas onto this dataset.\n",
    "SettAreas = gpd.read_file(r'Results/CumulativeSettlements.gpkg', layer=\n",
    "                          ''.join(['Cu', str(2015), '_Boundless'])) \n",
    "SettAreas['AREA2015'] = SettAreas['geometry'].area / 10**6\n",
    "SettAreas = pd.DataFrame(SettAreas).drop(columns='geometry') # We have settlement IDs, so no need to join spatially!\n",
    "\n",
    "\n",
    "for item in ReversedStudyYears_to2014:\n",
    "    print(\"Loading cumulative layer for year %s. %s\\n\" % (item, time.ctime()))\n",
    "    YearLayer = gpd.read_file(r'Results/CumulativeSettlements.gpkg', layer=''.join(['Cu', str(item), '_Boundless']))\n",
    "    print(\"Adding area field and converting to non-spatial dataframe. %s\\n\" % (time.ctime()))\n",
    "    AreaYearName = ''.join(['AREA', str(item)])\n",
    "    YearLayer[AreaYearName] = YearLayer['geometry'].area/ 10**6 \n",
    "    YearLayer = pd.DataFrame(YearLayer)[['Sett_ID', AreaYearName]]\n",
    "    print(\"Merging variables from %s onto our latest year (%s) via table join. %s\\n\" % (item, 2015, time.ctime()))\n",
    "    SettAreas = SettAreas.merge(YearLayer, how='left', on='Sett_ID')\n",
    "print(\"Done merging annualized areas onto latest year geometries. Saving to file. %s\\n\" % (time.ctime()))\n",
    "\n",
    "\n",
    "print(SettAreas.info())\n",
    "SettAreas.to_csv(os.path.join(ResultsFolder, 'Areas%sto%s.csv' % (1999, 2015)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "1d0879f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettAreas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a1cafcd",
   "metadata": {},
   "source": [
    "### 5.3 Repeat for Bounded dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "ad1dcf6f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subsetting to cumulative area for year: 1999. Thu Jan 26 11:26:05 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:26:05 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:26:33 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2000. Thu Jan 26 11:26:42 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:26:42 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:27:12 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2001. Thu Jan 26 11:27:20 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:27:20 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:27:52 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2002. Thu Jan 26 11:28:01 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:28:01 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:28:34 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2003. Thu Jan 26 11:28:44 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:28:44 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:29:19 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2004. Thu Jan 26 11:29:29 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:29:29 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:30:06 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2005. Thu Jan 26 11:30:15 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:30:15 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:30:54 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2006. Thu Jan 26 11:31:04 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:31:04 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:31:45 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2007. Thu Jan 26 11:31:55 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:31:55 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:32:38 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2008. Thu Jan 26 11:32:47 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:32:47 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:33:32 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2009. Thu Jan 26 11:33:42 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:33:42 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:34:29 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2010. Thu Jan 26 11:34:40 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:34:40 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:35:32 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2011. Thu Jan 26 11:35:42 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:35:42 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:36:31 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2012. Thu Jan 26 11:36:41 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:36:41 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:37:33 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2013. Thu Jan 26 11:37:42 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:37:42 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:38:36 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2014. Thu Jan 26 11:38:46 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:38:46 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:39:42 2023\n",
      "\n",
      "Subsetting to cumulative area for year: 2015. Thu Jan 26 11:39:52 2023\n",
      "\n",
      "Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. Thu Jan 26 11:39:52 2023\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_13496\\947833332.py:5: FutureWarning: Boolean inputs to the `inclusive` argument are deprecated in favour of `both` or `neither`.\n",
      "  YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to file. Thu Jan 26 11:40:51 2023\n",
      "\n",
      "Done with all years in set. Thu Jan 26 11:41:02 2023\n"
     ]
    }
   ],
   "source": [
    "# Bounded = gpd.read_file(r'Results/NonCumulativeSettlements.gpkg', layer='Settlements_Bounded')\n",
    "\n",
    "for item in AllStudyYears:\n",
    "    print('Subsetting to cumulative area for year: %s. %s\\n' % (item, time.ctime()))\n",
    "    YearSet = Bounded[Bounded['year'].between(1985, item, inclusive=True)] # Inclusive parameter means we include the years 1985 and \"item\" rather than only between them.\n",
    "    print('Dissolving so that each unique settlement (Bounded_ID) has a single cumulative WSFE feature. %s\\n' % time.ctime())\n",
    "    YearDissolve = YearSet.dissolve(by='Bounded_ID', \n",
    "                                        aggfunc={\"year\": \"max\", \"ADM_ID\":\"min\", \"Sett_ID\":\"min\"}, # Though ADM_ID and Sett_ID should be matching every time.\n",
    "                                        as_index=False)\n",
    "    print('Write to file. %s\\n' % time.ctime())\n",
    "    YearName = ''.join(['Cu', str(item), '_Bounded'])\n",
    "    YearDissolve.to_file(driver='GPKG', filename=r'Results/CumulativeSettlements.gpkg', layer=YearName)\n",
    "    del YearSet, YearDissolve\n",
    "print(\"Done with all years in set. %s\" % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "49eaeaeb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading cumulative layer for year 2014. Mon Feb 20 09:41:43 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:45 2023\n",
      "\n",
      "Merging variables from 2014 onto our latest year (2015) via table join. Mon Feb 20 09:41:45 2023\n",
      "\n",
      "Loading cumulative layer for year 2013. Mon Feb 20 09:41:45 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:47 2023\n",
      "\n",
      "Merging variables from 2013 onto our latest year (2015) via table join. Mon Feb 20 09:41:47 2023\n",
      "\n",
      "Loading cumulative layer for year 2012. Mon Feb 20 09:41:47 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:48 2023\n",
      "\n",
      "Merging variables from 2012 onto our latest year (2015) via table join. Mon Feb 20 09:41:49 2023\n",
      "\n",
      "Loading cumulative layer for year 2011. Mon Feb 20 09:41:49 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:50 2023\n",
      "\n",
      "Merging variables from 2011 onto our latest year (2015) via table join. Mon Feb 20 09:41:50 2023\n",
      "\n",
      "Loading cumulative layer for year 2010. Mon Feb 20 09:41:50 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:52 2023\n",
      "\n",
      "Merging variables from 2010 onto our latest year (2015) via table join. Mon Feb 20 09:41:52 2023\n",
      "\n",
      "Loading cumulative layer for year 2009. Mon Feb 20 09:41:52 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:54 2023\n",
      "\n",
      "Merging variables from 2009 onto our latest year (2015) via table join. Mon Feb 20 09:41:54 2023\n",
      "\n",
      "Loading cumulative layer for year 2008. Mon Feb 20 09:41:54 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:56 2023\n",
      "\n",
      "Merging variables from 2008 onto our latest year (2015) via table join. Mon Feb 20 09:41:56 2023\n",
      "\n",
      "Loading cumulative layer for year 2007. Mon Feb 20 09:41:56 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:57 2023\n",
      "\n",
      "Merging variables from 2007 onto our latest year (2015) via table join. Mon Feb 20 09:41:57 2023\n",
      "\n",
      "Loading cumulative layer for year 2006. Mon Feb 20 09:41:57 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:41:59 2023\n",
      "\n",
      "Merging variables from 2006 onto our latest year (2015) via table join. Mon Feb 20 09:41:59 2023\n",
      "\n",
      "Loading cumulative layer for year 2005. Mon Feb 20 09:41:59 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:01 2023\n",
      "\n",
      "Merging variables from 2005 onto our latest year (2015) via table join. Mon Feb 20 09:42:01 2023\n",
      "\n",
      "Loading cumulative layer for year 2004. Mon Feb 20 09:42:01 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:03 2023\n",
      "\n",
      "Merging variables from 2004 onto our latest year (2015) via table join. Mon Feb 20 09:42:03 2023\n",
      "\n",
      "Loading cumulative layer for year 2003. Mon Feb 20 09:42:03 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:04 2023\n",
      "\n",
      "Merging variables from 2003 onto our latest year (2015) via table join. Mon Feb 20 09:42:04 2023\n",
      "\n",
      "Loading cumulative layer for year 2002. Mon Feb 20 09:42:05 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:06 2023\n",
      "\n",
      "Merging variables from 2002 onto our latest year (2015) via table join. Mon Feb 20 09:42:06 2023\n",
      "\n",
      "Loading cumulative layer for year 2001. Mon Feb 20 09:42:06 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:08 2023\n",
      "\n",
      "Merging variables from 2001 onto our latest year (2015) via table join. Mon Feb 20 09:42:08 2023\n",
      "\n",
      "Loading cumulative layer for year 2000. Mon Feb 20 09:42:08 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:09 2023\n",
      "\n",
      "Merging variables from 2000 onto our latest year (2015) via table join. Mon Feb 20 09:42:10 2023\n",
      "\n",
      "Loading cumulative layer for year 1999. Mon Feb 20 09:42:10 2023\n",
      "\n",
      "Adding area field and converting to non-spatial dataframe. Mon Feb 20 09:42:11 2023\n",
      "\n",
      "Merging variables from 1999 onto our latest year (2015) via table join. Mon Feb 20 09:42:11 2023\n",
      "\n",
      "Done merging annualized areas onto latest year geometries. Saving to file. Mon Feb 20 09:42:11 2023\n",
      "\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23740 entries, 0 to 23739\n",
      "Data columns (total 21 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Bounded_ID  23740 non-null  int64  \n",
      " 1   year        23740 non-null  int64  \n",
      " 2   ADM_ID      23740 non-null  int64  \n",
      " 3   Sett_ID     23740 non-null  int64  \n",
      " 4   AREA2015    23740 non-null  float64\n",
      " 5   AREA2014    23129 non-null  float64\n",
      " 6   AREA2013    22692 non-null  float64\n",
      " 7   AREA2012    21844 non-null  float64\n",
      " 8   AREA2011    21527 non-null  float64\n",
      " 9   AREA2010    21150 non-null  float64\n",
      " 10  AREA2009    20662 non-null  float64\n",
      " 11  AREA2008    20245 non-null  float64\n",
      " 12  AREA2007    19886 non-null  float64\n",
      " 13  AREA2006    19536 non-null  float64\n",
      " 14  AREA2005    19175 non-null  float64\n",
      " 15  AREA2004    18846 non-null  float64\n",
      " 16  AREA2003    18419 non-null  float64\n",
      " 17  AREA2002    18042 non-null  float64\n",
      " 18  AREA2001    17748 non-null  float64\n",
      " 19  AREA2000    17287 non-null  float64\n",
      " 20  AREA1999    16531 non-null  float64\n",
      "dtypes: float64(17), int64(4)\n",
      "memory usage: 4.0 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "SettAreas = gpd.read_file(r'Results/CumulativeSettlements.gpkg', \n",
    "                          layer=''.join(['Cu', str(2015), '_Bounded']))\n",
    "SettAreas['AREA2015'] = SettAreas['geometry'].area / 10**6\n",
    "SettAreas = pd.DataFrame(SettAreas).drop(columns='geometry')\n",
    "\n",
    "\n",
    "for item in ReversedStudyYears_to2014:\n",
    "    print(\"Loading cumulative layer for year %s. %s\\n\" % (item, time.ctime()))\n",
    "    YearLayer = gpd.read_file(r'Results/CumulativeSettlements.gpkg', layer=''.join(['Cu', str(item), '_Bounded']))\n",
    "    print(\"Adding area field and converting to non-spatial dataframe. %s\\n\" % (time.ctime()))\n",
    "    AreaYearName = ''.join(['AREA', str(item)])\n",
    "    YearLayer[AreaYearName] = YearLayer['geometry'].area/ 10**6 \n",
    "    YearLayer = pd.DataFrame(YearLayer)[['Bounded_ID', AreaYearName]]\n",
    "    print(\"Merging variables from %s onto our latest year (%s) via table join. %s\\n\" % (item, 2015, time.ctime()))\n",
    "    SettAreas = SettAreas.merge(YearLayer, how='left', on='Bounded_ID')\n",
    "print(\"Done merging annualized areas onto latest year geometries. Saving to file. %s\\n\" % (time.ctime()))\n",
    "\n",
    "print(SettAreas.info())\n",
    "SettAreas.to_csv(os.path.join(ResultsFolder, 'Areas%sto%s_%s.csv' % (1999, 2015, 'Bounded')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "e7574cc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettAreas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "447ecaf8",
   "metadata": {},
   "source": [
    "### 5.4 One settlement geofile to rule them all. ...and in the Sett_ID bind them.\n",
    "The annualized values can be stored as distinct non-spatial dataframes. Their Sett_IDs will be used to join onto this geoversion with place names for the summary stats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "80315077",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   23490 non-null  int64   \n",
      " 1   ADM_ID    23490 non-null  int64   \n",
      " 2   geometry  23490 non-null  geometry\n",
      "dtypes: geometry(1), int64(2)\n",
      "memory usage: 550.7 KB\n",
      "None\n",
      "PROJCS[\"Africa_Albers_Equal_Area_Conic\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Albers_Conic_Equal_Area\"],PARAMETER[\"latitude_of_center\",0],PARAMETER[\"longitude_of_center\",25],PARAMETER[\"standard_parallel_1\",20],PARAMETER[\"standard_parallel_2\",-23],PARAMETER[\"false_easting\",0],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"ESRI\",\"102022\"]]\n"
     ]
    }
   ],
   "source": [
    "Settlements = gpd.read_file(r'Results/CumulativeSettlements.gpkg', \n",
    "                           layer=''.join(['Cu', str(2015), '_Boundless']))[['Sett_ID', 'ADM_ID', 'geometry']]\n",
    "print(Settlements.info())\n",
    "print(Settlements.crs)\n",
    "Settlements.to_file(driver='GPKG', \n",
    "                       filename=r'Results/SETTLEMENTS.gpkg', \n",
    "                       layer='SETTLEMENTS_equalarea')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "d12c8f85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Data columns (total 3 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   23490 non-null  int64   \n",
      " 1   ADM_ID    23490 non-null  int64   \n",
      " 2   geometry  23490 non-null  geometry\n",
      "dtypes: geometry(1), int64(2)\n",
      "memory usage: 550.7 KB\n",
      "None\n",
      "epsg:4326\n"
     ]
    }
   ],
   "source": [
    "# Saving all the final products as WGS84.\n",
    "Settlements_WGS = Settlements.to_crs(4326) \n",
    "print(Settlements_WGS.info())\n",
    "print(Settlements_WGS.crs)\n",
    "Settlements_WGS.to_file(driver='GPKG', \n",
    "                       filename=r'Results/SETTLEMENTS.gpkg', \n",
    "                       layer='SETTLEMENTS')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb8856d7",
   "metadata": {},
   "source": [
    "### 5.5 Buffer the area of the Boundless dataset's latest year to mask raster data in later sections.\n",
    "The Bounded dataset would also be fine for our purposes here. The buffer is dissolved to a single feature to be used for its total extents, which are identical between Bounded & Boundless datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "e8bb2910",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating buffer layer. Thu Jan 26 11:41:57 2023\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\miniconda3\\envs\\geo\\lib\\site-packages\\geopandas\\geodataframe.py:1472: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  super().__setitem__(key, value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished buffer layer creation. Thu Jan 26 11:55:50 2023\n",
      "Saved to file. Thu Jan 26 11:55:59 2023\n"
     ]
    }
   ],
   "source": [
    "# Create buffer layer(s) to use as maximum distance for Near joins.\n",
    "\n",
    "# Population buffer: 2km\n",
    "Distance = 2000 # The Africa Albers projection is in meters. Saving in this projection to use in later sections.\n",
    "\n",
    "print('Creating buffer layer. %s' % time.ctime())\n",
    "BufferLayer = Settlements[['Sett_ID', 'geometry']]\n",
    "BufferLayer['geometry'] = BufferLayer['geometry'].apply(\n",
    "    make_valid).buffer(Distance) # make_valid is a workaround for any null geometries.\n",
    "print('Finished buffer layer creation. %s' % time.ctime())\n",
    "BufferFileName1 = ''.join(['Buff', str(Distance), 'm_', str(2015)])\n",
    "BufferLayer.to_file(driver='GPKG', filename=r'Results/Catchment.gpkg', layer=BufferFileName1)\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "f1556d01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating buffer layer. Thu Jan 26 11:55:59 2023\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\miniconda3\\envs\\geo\\lib\\site-packages\\geopandas\\geodataframe.py:1472: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  super().__setitem__(key, value)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished buffer layer creation. Thu Jan 26 11:57:45 2023\n",
      "Saved to file. Thu Jan 26 11:57:55 2023\n"
     ]
    }
   ],
   "source": [
    "# Nighttime Lights buffer: 250m\n",
    "Distance = 250\n",
    "\n",
    "print('Creating buffer layer. %s' % time.ctime())\n",
    "BufferLayer = Settlements[['Sett_ID', 'geometry']]\n",
    "BufferLayer['geometry'] = BufferLayer['geometry'].apply(\n",
    "    make_valid).buffer(Distance) # make_valid is a workaround for any null geometries.\n",
    "print('Finished buffer layer creation. %s' % time.ctime())\n",
    "BufferFileName2 = ''.join(['Buff', str(Distance), 'm_', str(2015)])\n",
    "BufferLayer.to_file(driver='GPKG', filename=r'Results/Catchment.gpkg', layer=BufferFileName2)\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbef5b7f",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e33ce84a",
   "metadata": {},
   "source": [
    "## 6. PLACE NAMES\n",
    "Join urban place names from UCDB, Africapolis, and GeoNames onto the settlement vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b456bf69",
   "metadata": {},
   "source": [
    "### 6.1 Load placename datasets, filter, and project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "612f815f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Data columns (total 4 columns):\n",
      " #   Column    Non-Null Count  Dtype   \n",
      "---  ------    --------------  -----   \n",
      " 0   Sett_ID   23490 non-null  int64   \n",
      " 1   ADM_ID    23490 non-null  int64   \n",
      " 2   geometry  23490 non-null  geometry\n",
      " 3   Area2015  23490 non-null  float64 \n",
      "dtypes: float64(1), geometry(1), int64(2)\n",
      "memory usage: 734.2 KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 13135 entries, 0 to 13134\n",
      "Data columns (total 2 columns):\n",
      " #   Column     Non-Null Count  Dtype   \n",
      "---  ------     --------------  -----   \n",
      " 0   UCDB_Name  13135 non-null  object  \n",
      " 1   geometry   13135 non-null  geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 205.4+ KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 7720 entries, 0 to 7719\n",
      "Data columns (total 2 columns):\n",
      " #   Column     Non-Null Count  Dtype   \n",
      "---  ------     --------------  -----   \n",
      " 0   Afpl_Name  7720 non-null   object  \n",
      " 1   geometry   7720 non-null   geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 120.8+ KB\n",
      "<class 'geopandas.geodataframe.GeoDataFrame'>\n",
      "RangeIndex: 199390 entries, 0 to 199389\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count   Dtype   \n",
      "---  ------    --------------   -----   \n",
      " 0   GeoName   199390 non-null  object  \n",
      " 1   geometry  199390 non-null  geometry\n",
      "dtypes: geometry(1), object(1)\n",
      "memory usage: 3.0+ MB\n",
      "None None None None\n"
     ]
    }
   ],
   "source": [
    "# Anytime we use a spatial join or work with area, \n",
    "# my preference is to keep it in a planar, equal area, meters projection. So we'll load as the Africa Albers.\n",
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS_equalarea')\n",
    "Settlements['AREA2015'] = Settlements['geometry'].area / 10**6\n",
    "\n",
    "# Load, pull name field, rename, and reproject to match the catchments CRS.\n",
    "UCDB = gpd.read_file('PlaceName/GHS_STAT_UCDB2015MT_GLOBE_R2019A_V1_2.gpkg', \n",
    "                     layer=0)[['UC_NM_MN', 'geometry']].rename(\n",
    "    columns={\"UC_NM_MN\": \"UCDB_Name\"}).to_crs(\"ESRI:102022\")\n",
    "\n",
    "Africapolis = gpd.read_file('PlaceName/AFRICAPOLIS2020.shp')[['agglosName', 'geometry']].rename(\n",
    "    columns={\"agglosName\": \"Afpl_Name\"}).to_crs(\"ESRI:102022\")\n",
    "\n",
    "GeoNames = gpd.read_file('PlaceName/GeoNames.gpkg', \n",
    "                         layer=0)[['GeoName', 'geometry']].to_crs(\"ESRI:102022\")\n",
    "\n",
    "print(Settlements.info(), UCDB.info(), Africapolis.info(), GeoNames.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e934567",
   "metadata": {},
   "source": [
    "### 6.2 Join placenames onto settlements geodataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "106f2719",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We wrap it in pd.DataFrame() since the sjoin() is the last time we need the geometry.\n",
    "\n",
    "GeoNames = pd.DataFrame(gpd.sjoin_nearest(GeoNames, Settlements, \n",
    "                             how='left', distance_col=\"distGN\", max_distance=250, \n",
    "                             lsuffix=\"G3\", rsuffix=\"GN\")).drop(columns='geometry')\n",
    "Africapolis = pd.DataFrame(gpd.sjoin_nearest(Africapolis, Settlements, \n",
    "                             how='left', distance_col=\"distAF\", max_distance=250,\n",
    "                             lsuffix=\"G3\", rsuffix=\"Af\")).drop(columns='geometry')\n",
    "UCDB = pd.DataFrame(gpd.sjoin_nearest(UCDB, Settlements, \n",
    "                             how='left', distance_col=\"distUC\", max_distance=250,\n",
    "                             lsuffix=\"G3\", rsuffix=\"UC\")).drop(columns='geometry')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "2e2f0ee0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 199390 entries, 0 to 199389\n",
      "Data columns (total 6 columns):\n",
      " #   Column    Non-Null Count   Dtype  \n",
      "---  ------    --------------   -----  \n",
      " 0   GeoName   199390 non-null  object \n",
      " 1   index_GN  75 non-null      float64\n",
      " 2   Sett_ID   75 non-null      float64\n",
      " 3   ADM_ID    75 non-null      float64\n",
      " 4   Area2015  75 non-null      float64\n",
      " 5   distGN    75 non-null      float64\n",
      "dtypes: float64(5), object(1)\n",
      "memory usage: 10.6+ MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 7824 entries, 0 to 7719\n",
      "Data columns (total 6 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Afpl_Name  7824 non-null   object \n",
      " 1   index_Af   208 non-null    float64\n",
      " 2   Sett_ID    208 non-null    float64\n",
      " 3   ADM_ID     208 non-null    float64\n",
      " 4   Area2015   208 non-null    float64\n",
      " 5   distAF     208 non-null    float64\n",
      "dtypes: float64(5), object(1)\n",
      "memory usage: 427.9+ KB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 13201 entries, 0 to 13134\n",
      "Data columns (total 6 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   UCDB_Name  13201 non-null  object \n",
      " 1   index_UC   110 non-null    float64\n",
      " 2   Sett_ID    110 non-null    float64\n",
      " 3   ADM_ID     110 non-null    float64\n",
      " 4   Area2015   110 non-null    float64\n",
      " 5   distUC     110 non-null    float64\n",
      "dtypes: float64(5), object(1)\n",
      "memory usage: 721.9+ KB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(GeoNames.info())\n",
    "print(Africapolis.info())\n",
    "print(UCDB.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "96e1a75a",
   "metadata": {},
   "outputs": [],
   "source": [
    "alldatasets = [pd.DataFrame(Settlements).drop(columns='geometry'),\n",
    "               Africapolis[['Sett_ID', 'Afpl_Name', 'distAF']], \n",
    "               GeoNames[['Sett_ID', 'GeoName', 'distGN']],\n",
    "               UCDB[['Sett_ID', 'UCDB_Name', 'distUC']]]\n",
    "\n",
    "SettlementsNamed = reduce(lambda left,right: pd.merge(left,right,on=['Sett_ID'], how='left'), alldatasets)\n",
    "SettlementsNamed[['Afpl_Name', 'GeoName', 'UCDB_Name']] = SettlementsNamed[['Afpl_Name', 'GeoName', 'UCDB_Name']].fillna('UNK')\n",
    "\n",
    "# Replace NaN values with a countable distance.\n",
    "SettlementsNamed[['distAF', 'distGN', 'distUC']] = SettlementsNamed[['distAF', 'distGN', 'distUC']].fillna(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "3af429c4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23492 entries, 0 to 23491\n",
      "Data columns (total 9 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    23492 non-null  int64  \n",
      " 1   ADM_ID     23492 non-null  int64  \n",
      " 2   Area2015   23492 non-null  float64\n",
      " 3   Afpl_Name  23492 non-null  object \n",
      " 4   distAF     23492 non-null  float64\n",
      " 5   GeoName    23492 non-null  object \n",
      " 6   distGN     23492 non-null  float64\n",
      " 7   UCDB_Name  23492 non-null  object \n",
      " 8   distUC     23492 non-null  float64\n",
      "dtypes: float64(4), int64(2), object(3)\n",
      "memory usage: 1.8+ MB\n",
      "None\n",
      "       Sett_ID  ADM_ID  Area2015 Afpl_Name  distAF GeoName  distGN UCDB_Name  \\\n",
      "11524   266047      27  0.062039       UNK    -1.0     UNK    -1.0       UNK   \n",
      "14745   400586     130  0.016997       UNK    -1.0     UNK    -1.0       UNK   \n",
      "1108     29880     210  0.018697       UNK    -1.0     UNK    -1.0       UNK   \n",
      "18407   454527      54  0.013598       UNK    -1.0     UNK    -1.0       UNK   \n",
      "15152   404714     213  0.016997       UNK    -1.0     UNK    -1.0       UNK   \n",
      "1369     30217     222  0.047592       UNK    -1.0     UNK    -1.0       UNK   \n",
      "1839     37144     206  0.577901       UNK    -1.0     UNK    -1.0       UNK   \n",
      "1355     30197     222  0.032294       UNK    -1.0     UNK    -1.0       UNK   \n",
      "22326   528157     240  0.011898       UNK    -1.0     UNK    -1.0       UNK   \n",
      "1462     34297     181  0.005099       UNK    -1.0     UNK    -1.0       UNK   \n",
      "\n",
      "       distUC  \n",
      "11524    -1.0  \n",
      "14745    -1.0  \n",
      "1108     -1.0  \n",
      "18407    -1.0  \n",
      "15152    -1.0  \n",
      "1369     -1.0  \n",
      "1839     -1.0  \n",
      "1355     -1.0  \n",
      "22326    -1.0  \n",
      "1462     -1.0  \n"
     ]
    }
   ],
   "source": [
    "print(SettlementsNamed.info())\n",
    "print(SettlementsNamed.sample(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "8f92fab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "del UCDB, Africapolis, GeoNames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f2fc7cd",
   "metadata": {},
   "source": [
    "The near joins should have prevented duplication of rows, but if df1 intersects with two features in df2, it creates a new row. Two of our placenames sources are polygons, so there may be instances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "4f28f065",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>Area2015</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>distAF</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>distGN</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>distUC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9910</th>\n",
       "      <td>227047</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Saaba</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9911</th>\n",
       "      <td>227047</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11757</th>\n",
       "      <td>277234</td>\n",
       "      <td>265</td>\n",
       "      <td>3.837945</td>\n",
       "      <td>Cinkassi/Cinkanse [BFA]</td>\n",
       "      <td>0.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Cinkassé</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11758</th>\n",
       "      <td>277234</td>\n",
       "      <td>265</td>\n",
       "      <td>3.837945</td>\n",
       "      <td>Cinkassi/Cinkanse [TGO]</td>\n",
       "      <td>0.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>Cinkassé</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID  ADM_ID    Area2015                Afpl_Name  distAF  \\\n",
       "9910    227047     147  346.334596              Ouagadougou     0.0   \n",
       "9911    227047     147  346.334596              Ouagadougou     0.0   \n",
       "11757   277234     265    3.837945  Cinkassi/Cinkanse [BFA]     0.0   \n",
       "11758   277234     265    3.837945  Cinkassi/Cinkanse [TGO]     0.0   \n",
       "\n",
       "           GeoName  distGN    UCDB_Name  distUC  \n",
       "9910         Saaba     0.0  Ouagadougou     0.0  \n",
       "9911   Ouagadougou     0.0  Ouagadougou     0.0  \n",
       "11757          UNK    -1.0     Cinkassé     0.0  \n",
       "11758          UNK    -1.0     Cinkassé     0.0  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed[SettlementsNamed.duplicated('Sett_ID', keep=False)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "42d1b4b9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23490 entries, 0 to 23491\n",
      "Data columns (total 9 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    23490 non-null  int64  \n",
      " 1   ADM_ID     23490 non-null  int64  \n",
      " 2   Area2015   23490 non-null  float64\n",
      " 3   Afpl_Name  23490 non-null  object \n",
      " 4   distAF     23490 non-null  float64\n",
      " 5   GeoName    23490 non-null  object \n",
      " 6   distGN     23490 non-null  float64\n",
      " 7   UCDB_Name  23490 non-null  object \n",
      " 8   distUC     23490 non-null  float64\n",
      "dtypes: float64(4), int64(2), object(3)\n",
      "memory usage: 1.8+ MB\n"
     ]
    }
   ],
   "source": [
    "SettlementsNamed.drop_duplicates(subset=['Sett_ID'], inplace=True, keep='first')\n",
    "SettlementsNamed.info() # Range of entries should be the same as original Settlements file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65a16201",
   "metadata": {},
   "source": [
    "### 6.3 Reduce to single name column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "ad4c8be2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determine which source has a name geometrically closest to the settlement.\n",
    "# Since we switched NaN values to -1 earlier, we also resolved what happens in the event of a tie, \n",
    "# i.e. when more than one source is 0.0 meters from the settlement. It will take the value from the first column.\n",
    "SettlementsNamed['SettName'] = \"UNK\"\n",
    "SettlementsNamed['closest'] = SettlementsNamed[['distAF', 'distGN', 'distUC']].idxmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "f6551664",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>Area2015</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>distAF</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>distGN</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>distUC</th>\n",
       "      <th>SettName</th>\n",
       "      <th>closest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14191</th>\n",
       "      <td>388852</td>\n",
       "      <td>75</td>\n",
       "      <td>0.059490</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3088</th>\n",
       "      <td>66105</td>\n",
       "      <td>37</td>\n",
       "      <td>0.004249</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9379</th>\n",
       "      <td>215219</td>\n",
       "      <td>119</td>\n",
       "      <td>0.015297</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4938</th>\n",
       "      <td>117585</td>\n",
       "      <td>264</td>\n",
       "      <td>0.072238</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15635</th>\n",
       "      <td>409516</td>\n",
       "      <td>320</td>\n",
       "      <td>0.011898</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15632</th>\n",
       "      <td>409511</td>\n",
       "      <td>137</td>\n",
       "      <td>0.011898</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2260</th>\n",
       "      <td>51309</td>\n",
       "      <td>127</td>\n",
       "      <td>0.028895</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17761</th>\n",
       "      <td>446872</td>\n",
       "      <td>257</td>\n",
       "      <td>0.003399</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10077</th>\n",
       "      <td>227336</td>\n",
       "      <td>290</td>\n",
       "      <td>0.007649</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11461</th>\n",
       "      <td>265911</td>\n",
       "      <td>187</td>\n",
       "      <td>0.045042</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3385</th>\n",
       "      <td>83528</td>\n",
       "      <td>269</td>\n",
       "      <td>0.005949</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4671</th>\n",
       "      <td>117227</td>\n",
       "      <td>159</td>\n",
       "      <td>0.028895</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13522</th>\n",
       "      <td>319194</td>\n",
       "      <td>192</td>\n",
       "      <td>0.022096</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10165</th>\n",
       "      <td>227451</td>\n",
       "      <td>231</td>\n",
       "      <td>0.019547</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6411</th>\n",
       "      <td>159734</td>\n",
       "      <td>74</td>\n",
       "      <td>0.012748</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23102</th>\n",
       "      <td>581370</td>\n",
       "      <td>81</td>\n",
       "      <td>0.021246</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18356</th>\n",
       "      <td>454344</td>\n",
       "      <td>23</td>\n",
       "      <td>0.016997</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7667</th>\n",
       "      <td>186524</td>\n",
       "      <td>102</td>\n",
       "      <td>0.008499</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>16233</td>\n",
       "      <td>197</td>\n",
       "      <td>0.028895</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12093</th>\n",
       "      <td>277946</td>\n",
       "      <td>285</td>\n",
       "      <td>0.001700</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID  ADM_ID  Area2015 Afpl_Name  distAF GeoName  distGN UCDB_Name  \\\n",
       "14191   388852      75  0.059490       UNK    -1.0     UNK    -1.0       UNK   \n",
       "3088     66105      37  0.004249       UNK    -1.0     UNK    -1.0       UNK   \n",
       "9379    215219     119  0.015297       UNK    -1.0     UNK    -1.0       UNK   \n",
       "4938    117585     264  0.072238       UNK    -1.0     UNK    -1.0       UNK   \n",
       "15635   409516     320  0.011898       UNK    -1.0     UNK    -1.0       UNK   \n",
       "15632   409511     137  0.011898       UNK    -1.0     UNK    -1.0       UNK   \n",
       "2260     51309     127  0.028895       UNK    -1.0     UNK    -1.0       UNK   \n",
       "17761   446872     257  0.003399       UNK    -1.0     UNK    -1.0       UNK   \n",
       "10077   227336     290  0.007649       UNK    -1.0     UNK    -1.0       UNK   \n",
       "11461   265911     187  0.045042       UNK    -1.0     UNK    -1.0       UNK   \n",
       "3385     83528     269  0.005949       UNK    -1.0     UNK    -1.0       UNK   \n",
       "4671    117227     159  0.028895       UNK    -1.0     UNK    -1.0       UNK   \n",
       "13522   319194     192  0.022096       UNK    -1.0     UNK    -1.0       UNK   \n",
       "10165   227451     231  0.019547       UNK    -1.0     UNK    -1.0       UNK   \n",
       "6411    159734      74  0.012748       UNK    -1.0     UNK    -1.0       UNK   \n",
       "23102   581370      81  0.021246       UNK    -1.0     UNK    -1.0       UNK   \n",
       "18356   454344      23  0.016997       UNK    -1.0     UNK    -1.0       UNK   \n",
       "7667    186524     102  0.008499       UNK    -1.0     UNK    -1.0       UNK   \n",
       "871      16233     197  0.028895       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12093   277946     285  0.001700       UNK    -1.0     UNK    -1.0       UNK   \n",
       "\n",
       "       distUC SettName closest  \n",
       "14191    -1.0      UNK  distAF  \n",
       "3088     -1.0      UNK  distAF  \n",
       "9379     -1.0      UNK  distAF  \n",
       "4938     -1.0      UNK  distAF  \n",
       "15635    -1.0      UNK  distAF  \n",
       "15632    -1.0      UNK  distAF  \n",
       "2260     -1.0      UNK  distAF  \n",
       "17761    -1.0      UNK  distAF  \n",
       "10077    -1.0      UNK  distAF  \n",
       "11461    -1.0      UNK  distAF  \n",
       "3385     -1.0      UNK  distAF  \n",
       "4671     -1.0      UNK  distAF  \n",
       "13522    -1.0      UNK  distAF  \n",
       "10165    -1.0      UNK  distAF  \n",
       "6411     -1.0      UNK  distAF  \n",
       "23102    -1.0      UNK  distAF  \n",
       "18356    -1.0      UNK  distAF  \n",
       "7667     -1.0      UNK  distAF  \n",
       "871      -1.0      UNK  distAF  \n",
       "12093    -1.0      UNK  distAF  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed.sample(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "21fcb748",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a single name column where non-named settlements are \"UNK\" but all others use one of the three name sources.\n",
    "SettlementsNamed.loc[\n",
    "    SettlementsNamed['closest'] == \"distAF\", \n",
    "    'SettName'] = SettlementsNamed['Afpl_Name']\n",
    "\n",
    "SettlementsNamed.loc[\n",
    "    SettlementsNamed['closest'] == \"distUC\", \n",
    "    'SettName'] = SettlementsNamed['UCDB_Name']\n",
    "\n",
    "SettlementsNamed.loc[\n",
    "    SettlementsNamed['closest'] == \"distGN\", \n",
    "    'SettName'] = SettlementsNamed['GeoName']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "b754859a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>Area2015</th>\n",
       "      <th>Afpl_Name</th>\n",
       "      <th>distAF</th>\n",
       "      <th>GeoName</th>\n",
       "      <th>distGN</th>\n",
       "      <th>UCDB_Name</th>\n",
       "      <th>distUC</th>\n",
       "      <th>SettName</th>\n",
       "      <th>closest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7582</th>\n",
       "      <td>185243</td>\n",
       "      <td>63</td>\n",
       "      <td>0.011048</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20822</th>\n",
       "      <td>474698</td>\n",
       "      <td>134</td>\n",
       "      <td>0.014448</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16942</th>\n",
       "      <td>417578</td>\n",
       "      <td>20</td>\n",
       "      <td>0.046742</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6827</th>\n",
       "      <td>166088</td>\n",
       "      <td>276</td>\n",
       "      <td>0.098583</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10261</th>\n",
       "      <td>227649</td>\n",
       "      <td>255</td>\n",
       "      <td>0.062889</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20211</th>\n",
       "      <td>468259</td>\n",
       "      <td>306</td>\n",
       "      <td>0.003399</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18770</th>\n",
       "      <td>457220</td>\n",
       "      <td>121</td>\n",
       "      <td>0.006799</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2882</th>\n",
       "      <td>55834</td>\n",
       "      <td>37</td>\n",
       "      <td>0.000850</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5911</th>\n",
       "      <td>138901</td>\n",
       "      <td>41</td>\n",
       "      <td>0.021246</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10896</th>\n",
       "      <td>247572</td>\n",
       "      <td>346</td>\n",
       "      <td>0.075637</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16192</th>\n",
       "      <td>415812</td>\n",
       "      <td>334</td>\n",
       "      <td>0.004249</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12157</th>\n",
       "      <td>288592</td>\n",
       "      <td>170</td>\n",
       "      <td>0.004249</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16526</th>\n",
       "      <td>416522</td>\n",
       "      <td>347</td>\n",
       "      <td>0.006799</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15621</th>\n",
       "      <td>409493</td>\n",
       "      <td>39</td>\n",
       "      <td>0.005949</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1796</th>\n",
       "      <td>37097</td>\n",
       "      <td>165</td>\n",
       "      <td>0.194617</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12969</th>\n",
       "      <td>310059</td>\n",
       "      <td>163</td>\n",
       "      <td>0.000850</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8641</th>\n",
       "      <td>203882</td>\n",
       "      <td>136</td>\n",
       "      <td>0.101983</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22228</th>\n",
       "      <td>527936</td>\n",
       "      <td>31</td>\n",
       "      <td>0.006799</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16947</th>\n",
       "      <td>417654</td>\n",
       "      <td>334</td>\n",
       "      <td>0.003399</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21070</th>\n",
       "      <td>475154</td>\n",
       "      <td>3</td>\n",
       "      <td>0.016997</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>UNK</td>\n",
       "      <td>distAF</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID  ADM_ID  Area2015 Afpl_Name  distAF GeoName  distGN UCDB_Name  \\\n",
       "7582    185243      63  0.011048       UNK    -1.0     UNK    -1.0       UNK   \n",
       "20822   474698     134  0.014448       UNK    -1.0     UNK    -1.0       UNK   \n",
       "16942   417578      20  0.046742       UNK    -1.0     UNK    -1.0       UNK   \n",
       "6827    166088     276  0.098583       UNK    -1.0     UNK    -1.0       UNK   \n",
       "10261   227649     255  0.062889       UNK    -1.0     UNK    -1.0       UNK   \n",
       "20211   468259     306  0.003399       UNK    -1.0     UNK    -1.0       UNK   \n",
       "18770   457220     121  0.006799       UNK    -1.0     UNK    -1.0       UNK   \n",
       "2882     55834      37  0.000850       UNK    -1.0     UNK    -1.0       UNK   \n",
       "5911    138901      41  0.021246       UNK    -1.0     UNK    -1.0       UNK   \n",
       "10896   247572     346  0.075637       UNK    -1.0     UNK    -1.0       UNK   \n",
       "16192   415812     334  0.004249       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12157   288592     170  0.004249       UNK    -1.0     UNK    -1.0       UNK   \n",
       "16526   416522     347  0.006799       UNK    -1.0     UNK    -1.0       UNK   \n",
       "15621   409493      39  0.005949       UNK    -1.0     UNK    -1.0       UNK   \n",
       "1796     37097     165  0.194617       UNK    -1.0     UNK    -1.0       UNK   \n",
       "12969   310059     163  0.000850       UNK    -1.0     UNK    -1.0       UNK   \n",
       "8641    203882     136  0.101983       UNK    -1.0     UNK    -1.0       UNK   \n",
       "22228   527936      31  0.006799       UNK    -1.0     UNK    -1.0       UNK   \n",
       "16947   417654     334  0.003399       UNK    -1.0     UNK    -1.0       UNK   \n",
       "21070   475154       3  0.016997       UNK    -1.0     UNK    -1.0       UNK   \n",
       "\n",
       "       distUC SettName closest  \n",
       "7582     -1.0      UNK  distAF  \n",
       "20822    -1.0      UNK  distAF  \n",
       "16942    -1.0      UNK  distAF  \n",
       "6827     -1.0      UNK  distAF  \n",
       "10261    -1.0      UNK  distAF  \n",
       "20211    -1.0      UNK  distAF  \n",
       "18770    -1.0      UNK  distAF  \n",
       "2882     -1.0      UNK  distAF  \n",
       "5911     -1.0      UNK  distAF  \n",
       "10896    -1.0      UNK  distAF  \n",
       "16192    -1.0      UNK  distAF  \n",
       "12157    -1.0      UNK  distAF  \n",
       "16526    -1.0      UNK  distAF  \n",
       "15621    -1.0      UNK  distAF  \n",
       "1796     -1.0      UNK  distAF  \n",
       "12969    -1.0      UNK  distAF  \n",
       "8641     -1.0      UNK  distAF  \n",
       "22228    -1.0      UNK  distAF  \n",
       "16947    -1.0      UNK  distAF  \n",
       "21070    -1.0      UNK  distAF  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SettlementsNamed.sample(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c748e23",
   "metadata": {},
   "source": [
    "### 6.4 Make sure place name is unique by stripping smaller localities of duplicated names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "a0ec2f07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of named settlements: 274\n",
      "Number of named settlements where name is duplicated at least once: 186\n"
     ]
    }
   ],
   "source": [
    "Dupes = SettlementsNamed[ \n",
    "    (SettlementsNamed['SettName'] != 'UNK') & \n",
    "    (SettlementsNamed.duplicated('SettName', keep=False)) ] # keep=False is necessary to retain *all* duplicates, not just first or last in each group.\n",
    "\n",
    "print(\"Number of named settlements: %s\" % SettlementsNamed['SettName'].str.contains('UNK').value_counts()[False])\n",
    "print(\"Number of named settlements where name is duplicated at least once: %s\" % len(Dupes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "030d7d5a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       Sett_ID  ADM_ID    Area2015                Afpl_Name  distAF  \\\n",
      "7            8      14   16.495685                  Banfora     0.0   \n",
      "13929   388531      18    0.421528                      UNK    -1.0   \n",
      "16061   415490      20    1.024925               Barsalogho     0.0   \n",
      "2177     51196      37   88.690864           Bobo-Dioulasso     0.0   \n",
      "22184   527798      38    1.874780                  Bogande     0.0   \n",
      "5268    129734      43    3.492904                   Boromo     0.0   \n",
      "15003   404331      52    2.207073                   Bousse     0.0   \n",
      "11757   277234     265    3.837945  Cinkassi/Cinkanse [BFA]     0.0   \n",
      "2170     51189      59    2.212172                    Dande     0.0   \n",
      "5552    137698      64   12.117232                 Dedougou     0.0   \n",
      "7008    180084      70    0.961186                    Didyr     0.0   \n",
      "20499   474186      76    3.520099                    Djibo     0.0   \n",
      "22880   580905      81    4.186386                     Dori     0.0   \n",
      "22879   580904      88    0.578751              Falagountou     0.0   \n",
      "3543     87014      96    7.829714                    Gaoua     0.0   \n",
      "10723   247086      97    5.408477                  Garango     0.0   \n",
      "23340   597212     108    2.142484              Gorom-Gorom     0.0   \n",
      "14664   400481     171    2.184977               Guiedougou     0.0   \n",
      "4339    102549     116    9.152938                   Hounde     0.0   \n",
      "8055    202963     117    1.967414                  Imasgho     0.0   \n",
      "8058    202966     136    1.035123                    Kindi     0.0   \n",
      "9341    215150     142    5.436522               Kombissiri     0.0   \n",
      "13625   351201     146    2.007357                Kompienga     0.0   \n",
      "17117   445141     150    2.199425                Kongoussi     0.0   \n",
      "8057    202965     157   30.684014                Koudougou     0.0   \n",
      "4645    117196     159    3.594037                    Kouka     0.0   \n",
      "12674   309527     123    8.753506                  Koupela     0.0   \n",
      "7200    180351     168    0.833708                     Kyon     0.0   \n",
      "3544     87015     180    1.098013                      UNK    -1.0   \n",
      "11344   265667     187    4.363155                    Manga     0.0   \n",
      "16714   417069     348    0.047592             Namissiguima     0.0   \n",
      "13743   358776     179    0.499715               Namponkore     0.0   \n",
      "2176     51195     316    2.598856                  Nianaba     0.0   \n",
      "2            3     209    6.496291               Niangoloko     0.0   \n",
      "13922   388524     215    5.520658                    Nouna     0.0   \n",
      "9910    227047     147  346.334596              Ouagadougou     0.0   \n",
      "16060   415489     242    1.398861                  Pissila     0.0   \n",
      "12676   309529     249   10.984375                Pouytenga     0.0   \n",
      "7009    180085     252    7.373342                      Reo     0.0   \n",
      "17112   445136     254    0.389234                    Rouko     0.0   \n",
      "7769    188381     268    2.814720                   Sapouy     0.0   \n",
      "13          14     274    1.491495                      UNK    -1.0   \n",
      "4638    117189     280    7.262861                  Solenzo     0.0   \n",
      "9902    227039     290    3.977321         Tanghin-Dassouri     0.0   \n",
      "6583    165722     307    0.852405                       To     0.0   \n",
      "14666   400483     313    3.594037                   Tougan     0.0   \n",
      "21689   508698     315    1.067418                      UNK    -1.0   \n",
      "8519    203707     157    0.222662                    Villy     0.0   \n",
      "12076   277909     144    0.469120                 Vohogdin     0.0   \n",
      "21302   501405     336    8.655773                  Ziniare     0.0   \n",
      "12177   294181     342    5.605643                   Zorgho     0.0   \n",
      "7011    180087     252    1.262884                    Zoula     0.0   \n",
      "\n",
      "                GeoName      distGN       UCDB_Name  distUC  \\\n",
      "7               Banfora    0.000000         Banfora     0.0   \n",
      "13929            Barani    0.000000          Barani     0.0   \n",
      "16061               UNK   -1.000000      Barsalogho     0.0   \n",
      "2177     Bobo-Dioulasso    0.000000  Bobo-Dioulasso     0.0   \n",
      "22184           Bogandé    0.000000         Bogandé     0.0   \n",
      "5268             Boromo    0.000000             UNK    -1.0   \n",
      "15003            Boussé    0.000000             UNK    -1.0   \n",
      "11757               UNK   -1.000000        Cinkassé     0.0   \n",
      "2170                UNK   -1.000000             UNK    -1.0   \n",
      "5552           Dédougou    0.000000        Dédougou     0.0   \n",
      "7008                UNK   -1.000000             UNK    -1.0   \n",
      "20499             Djibo    0.000000           Djibo     0.0   \n",
      "22880              Dori    0.000000            Dori     0.0   \n",
      "22879               UNK   -1.000000             UNK    -1.0   \n",
      "3543                UNK   -1.000000           Gaoua     0.0   \n",
      "10723           Garango    0.000000         Garango     0.0   \n",
      "23340       Gorom-Gorom    0.000000     Gorom-Gorom     0.0   \n",
      "14664               UNK   -1.000000             UNK    -1.0   \n",
      "4339                UNK   -1.000000          Houndé     0.0   \n",
      "8055                UNK   -1.000000             UNK    -1.0   \n",
      "8058                UNK   -1.000000             UNK    -1.0   \n",
      "9341         Kombissiri    0.000000      Kombissiri     0.0   \n",
      "13625               UNK   -1.000000       Kompienga     0.0   \n",
      "17117         Kongoussi  249.051436       Kongoussi     0.0   \n",
      "8057          Koudougou    0.000000       Koudougou     0.0   \n",
      "4645                UNK   -1.000000             UNK    -1.0   \n",
      "12674           Koupéla    0.000000         Koupéla     0.0   \n",
      "7200                UNK   -1.000000             UNK    -1.0   \n",
      "3544                UNK   -1.000000        Loropéni     0.0   \n",
      "11344             Manga    0.000000             UNK    -1.0   \n",
      "16714               UNK   -1.000000             UNK    -1.0   \n",
      "13743               UNK   -1.000000             UNK    -1.0   \n",
      "2176                UNK   -1.000000             UNK    -1.0   \n",
      "2                   UNK   -1.000000      Niangoloko     0.0   \n",
      "13922             Nouna    0.000000           Nouna     0.0   \n",
      "9910              Saaba    0.000000     Ouagadougou     0.0   \n",
      "16060               UNK   -1.000000         Pissila     0.0   \n",
      "12676               UNK   -1.000000       Pouytenga     0.0   \n",
      "7009                Réo    0.000000             UNK    -1.0   \n",
      "17112               UNK   -1.000000             UNK    -1.0   \n",
      "7769             Sapouy    4.095599          Sapouy     0.0   \n",
      "13                  UNK   -1.000000    Sidéradougou     0.0   \n",
      "4638            Salanso    0.000000         Solenzo     0.0   \n",
      "9902   Tanghin-Dassouri    0.000000             UNK    -1.0   \n",
      "6583                UNK   -1.000000             UNK    -1.0   \n",
      "14666            Tougan    0.000000          Tougan     0.0   \n",
      "21689               UNK   -1.000000        Tougouri     0.0   \n",
      "8519                UNK   -1.000000             UNK    -1.0   \n",
      "12076               UNK   -1.000000             UNK    -1.0   \n",
      "21302           Ziniaré    0.000000         Ziniaré     0.0   \n",
      "12177             Zorgo    0.000000             UNK    -1.0   \n",
      "7011                UNK   -1.000000             UNK    -1.0   \n",
      "\n",
      "                      SettName closest  \n",
      "7                      Banfora  distAF  \n",
      "13929                   Barani  distGN  \n",
      "16061               Barsalogho  distAF  \n",
      "2177            Bobo-Dioulasso  distAF  \n",
      "22184                  Bogande  distAF  \n",
      "5268                    Boromo  distAF  \n",
      "15003                   Bousse  distAF  \n",
      "11757  Cinkassi/Cinkanse [BFA]  distAF  \n",
      "2170                     Dande  distAF  \n",
      "5552                  Dedougou  distAF  \n",
      "7008                     Didyr  distAF  \n",
      "20499                    Djibo  distAF  \n",
      "22880                     Dori  distAF  \n",
      "22879              Falagountou  distAF  \n",
      "3543                     Gaoua  distAF  \n",
      "10723                  Garango  distAF  \n",
      "23340              Gorom-Gorom  distAF  \n",
      "14664               Guiedougou  distAF  \n",
      "4339                    Hounde  distAF  \n",
      "8055                   Imasgho  distAF  \n",
      "8058                     Kindi  distAF  \n",
      "9341                Kombissiri  distAF  \n",
      "13625                Kompienga  distAF  \n",
      "17117                Kongoussi  distGN  \n",
      "8057                 Koudougou  distAF  \n",
      "4645                     Kouka  distAF  \n",
      "12674                  Koupela  distAF  \n",
      "7200                      Kyon  distAF  \n",
      "3544                  Loropéni  distUC  \n",
      "11344                    Manga  distAF  \n",
      "16714             Namissiguima  distAF  \n",
      "13743               Namponkore  distAF  \n",
      "2176                   Nianaba  distAF  \n",
      "2                   Niangoloko  distAF  \n",
      "13922                    Nouna  distAF  \n",
      "9910               Ouagadougou  distAF  \n",
      "16060                  Pissila  distAF  \n",
      "12676                Pouytenga  distAF  \n",
      "7009                       Reo  distAF  \n",
      "17112                    Rouko  distAF  \n",
      "7769                    Sapouy  distGN  \n",
      "13                Sidéradougou  distUC  \n",
      "4638                   Solenzo  distAF  \n",
      "9902          Tanghin-Dassouri  distAF  \n",
      "6583                        To  distAF  \n",
      "14666                   Tougan  distAF  \n",
      "21689                 Tougouri  distUC  \n",
      "8519                     Villy  distAF  \n",
      "12076                 Vohogdin  distAF  \n",
      "21302                  Ziniare  distAF  \n",
      "12177                   Zorgho  distAF  \n",
      "7011                     Zoula  distAF  \n"
     ]
    }
   ],
   "source": [
    "Largest = Dupes.loc[Dupes.groupby([\"SettName\"])[\"AREA2015\"].idxmax()]\n",
    "print(Largest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "82bd26e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter to settlements which have a duplicated name and are not the largest of those with that name, then replace with UNK.\n",
    "SettlementsNamed.loc[(~SettlementsNamed.Sett_ID.isin(Largest.Sett_ID)) \n",
    "                     & (SettlementsNamed.Sett_ID.isin(Dupes.Sett_ID)), \n",
    "                     'SettName'] = 'UNK'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "28bddc71",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of named settlements: 140\n",
      "Number of named settlements where name is duplicated at least once: 0\n"
     ]
    }
   ],
   "source": [
    "# Second number should now be zero.\n",
    "\n",
    "print(\"Number of named settlements: %s\" % SettlementsNamed['SettName'].str.contains('UNK').value_counts()[False])\n",
    "print(\"Number of named settlements where name is duplicated at least once: %s\" % len(SettlementsNamed[ \n",
    "    (SettlementsNamed['SettName'] != 'UNK') & \n",
    "    (SettlementsNamed.duplicated('SettName', keep=False)) ]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "885dd231",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23490 entries, 0 to 23491\n",
      "Data columns (total 11 columns):\n",
      " #   Column     Non-Null Count  Dtype  \n",
      "---  ------     --------------  -----  \n",
      " 0   Sett_ID    23490 non-null  int64  \n",
      " 1   ADM_ID     23490 non-null  int64  \n",
      " 2   Area2015   23490 non-null  float64\n",
      " 3   Afpl_Name  23490 non-null  object \n",
      " 4   distAF     23490 non-null  float64\n",
      " 5   GeoName    23490 non-null  object \n",
      " 6   distGN     23490 non-null  float64\n",
      " 7   UCDB_Name  23490 non-null  object \n",
      " 8   distUC     23490 non-null  float64\n",
      " 9   SettName   23490 non-null  object \n",
      " 10  closest    23490 non-null  object \n",
      "dtypes: float64(4), int64(2), object(5)\n",
      "memory usage: 2.7+ MB\n",
      "None        Sett_ID  ADM_ID  Area2015                         Afpl_Name  \\\n",
      "13626   351202     233  1.804242                          Nadiagou   \n",
      "15141   404684     213  0.720677                               UNK   \n",
      "8058    202966     136  1.035123                             Kindi   \n",
      "17117   445141     150  2.199425                         Kongoussi   \n",
      "8054    202962     140  3.282990                         Kokologho   \n",
      "20501   474189       3  0.503114                           Arbinda   \n",
      "11759   277235     220  1.550985                          Ouargaye   \n",
      "11          12     226  0.345041                               UNK   \n",
      "3408     84247      24  1.630022                             Batie   \n",
      "11342   265665     105  2.036252                        Gonboussou   \n",
      "7008    180084      70  0.961186                             Didyr   \n",
      "13922   388524     215  5.520658                             Nouna   \n",
      "8969    204638     140  0.250707                               UNK   \n",
      "13          14     274  1.491495                               UNK   \n",
      "9341    215150     142  5.436522                        Kombissiri   \n",
      "16060   415489     242  1.398861                           Pissila   \n",
      "7011    180087     252  1.262884                             Zoula   \n",
      "13743   358776     179  0.499715                        Namponkore   \n",
      "23421   597575     229  0.210764                               UNK   \n",
      "10540   236585     301  0.178470  Manyoro/Kolonia-Tangassogo [GHA]   \n",
      "\n",
      "           distAF     GeoName      distGN     UCDB_Name  distUC  \\\n",
      "13626    0.000000         UNK   -1.000000           UNK    -1.0   \n",
      "15141   -1.000000        Niou    0.000000           UNK    -1.0   \n",
      "8058     0.000000         UNK   -1.000000           UNK    -1.0   \n",
      "17117    0.000000   Kongoussi  249.051436     Kongoussi     0.0   \n",
      "8054     0.000000    Kokologo    0.000000           UNK    -1.0   \n",
      "20501    0.000000         UNK   -1.000000       Arbinda     0.0   \n",
      "11759    0.000000    Ouargaye    0.000000           UNK    -1.0   \n",
      "11      -1.000000         UNK   -1.000000           Ouo     0.0   \n",
      "3408     0.000000       Batié    0.124172           UNK    -1.0   \n",
      "11342    0.000000         UNK   -1.000000           UNK    -1.0   \n",
      "7008     0.000000         UNK   -1.000000           UNK    -1.0   \n",
      "13922    0.000000       Nouna    0.000000         Nouna     0.0   \n",
      "8969    -1.000000    Pitmoaga  175.820310           UNK    -1.0   \n",
      "13      -1.000000         UNK   -1.000000  Sidéradougou     0.0   \n",
      "9341     0.000000  Kombissiri    0.000000    Kombissiri     0.0   \n",
      "16060    0.000000         UNK   -1.000000       Pissila     0.0   \n",
      "7011     0.000000         UNK   -1.000000           UNK    -1.0   \n",
      "13743    0.000000         UNK   -1.000000           UNK    -1.0   \n",
      "23421   -1.000000       Oursi    0.000000           UNK    -1.0   \n",
      "10540  207.117679         UNK   -1.000000           UNK    -1.0   \n",
      "\n",
      "                               SettName closest  \n",
      "13626                          Nadiagou  distAF  \n",
      "15141                              Niou  distGN  \n",
      "8058                              Kindi  distAF  \n",
      "17117                         Kongoussi  distGN  \n",
      "8054                          Kokologho  distAF  \n",
      "20501                           Arbinda  distAF  \n",
      "11759                          Ouargaye  distAF  \n",
      "11                                  Ouo  distUC  \n",
      "3408                              Batié  distGN  \n",
      "11342                        Gonboussou  distAF  \n",
      "7008                              Didyr  distAF  \n",
      "13922                             Nouna  distAF  \n",
      "8969                           Pitmoaga  distGN  \n",
      "13                         Sidéradougou  distUC  \n",
      "9341                         Kombissiri  distAF  \n",
      "16060                           Pissila  distAF  \n",
      "7011                              Zoula  distAF  \n",
      "13743                        Namponkore  distAF  \n",
      "23421                             Oursi  distGN  \n",
      "10540  Manyoro/Kolonia-Tangassogo [GHA]  distAF  \n"
     ]
    }
   ],
   "source": [
    "print(SettlementsNamed.info(), SettlementsNamed[SettlementsNamed['SettName'] != \"UNK\"].sample(20))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "0d202603",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop extra columns and save to file.\n",
    "SettlementsNamed = SettlementsNamed[['Sett_ID', 'SettName']]\n",
    "SettlementsNamed.to_csv(r'Results/PlaceNames.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "db1e4aa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "del SettlementsNamed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0c1c3d6",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d8d7b6d",
   "metadata": {},
   "source": [
    "## 7. CREATE FRAGMENTATION INDEX\n",
    "We are determining what percentage of a settlement's area lies outside of its administrative zone each year.\n",
    "The index is a range of 0 to 100, i.e. the percent of the settlement area which is fragmented.\n",
    "\n",
    "For each Sett_ID:\n",
    "((Area of Boundless settlement - Area of largest Bounded settlement feature) / Area of Boundless settlement) * 100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fa7537f",
   "metadata": {},
   "source": [
    "### 7.1 Load boundless and bounded cumulative settlements and clean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "7b9bdf01",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded Boundless dataset, whose settlements will be used as the index of the Fragmentation Index dataset. Thu Jan 26 13:47:43 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Data columns (total 21 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  23490 non-null  int64  \n",
      " 1   Sett_ID     23490 non-null  int64  \n",
      " 2   year        23490 non-null  int64  \n",
      " 3   ADM_ID      23490 non-null  int64  \n",
      " 4   Area2015    23490 non-null  float64\n",
      " 5   Area2014    22886 non-null  float64\n",
      " 6   Area2013    22457 non-null  float64\n",
      " 7   Area2012    21629 non-null  float64\n",
      " 8   Area2011    21316 non-null  float64\n",
      " 9   Area2010    20948 non-null  float64\n",
      " 10  Area2009    20471 non-null  float64\n",
      " 11  Area2008    20062 non-null  float64\n",
      " 12  Area2007    19712 non-null  float64\n",
      " 13  Area2006    19369 non-null  float64\n",
      " 14  Area2005    19010 non-null  float64\n",
      " 15  Area2004    18683 non-null  float64\n",
      " 16  Area2003    18259 non-null  float64\n",
      " 17  Area2002    17884 non-null  float64\n",
      " 18  Area2001    17592 non-null  float64\n",
      " 19  Area2000    17134 non-null  float64\n",
      " 20  Area1999    16385 non-null  float64\n",
      "dtypes: float64(17), int64(4)\n",
      "memory usage: 3.8 MB\n",
      "None\n",
      "Loaded Bounded dataset, which will factor into the fragmentation calculation. Thu Jan 26 13:47:44 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 23740 entries, 0 to 23739\n",
      "Data columns (total 22 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  23740 non-null  int64  \n",
      " 1   Bounded_ID  23740 non-null  int64  \n",
      " 2   year        23740 non-null  int64  \n",
      " 3   ADM_ID      23740 non-null  int64  \n",
      " 4   Sett_ID     23740 non-null  int64  \n",
      " 5   Area2015    23740 non-null  float64\n",
      " 6   Area2014    23129 non-null  float64\n",
      " 7   Area2013    22692 non-null  float64\n",
      " 8   Area2012    21844 non-null  float64\n",
      " 9   Area2011    21527 non-null  float64\n",
      " 10  Area2010    21150 non-null  float64\n",
      " 11  Area2009    20662 non-null  float64\n",
      " 12  Area2008    20245 non-null  float64\n",
      " 13  Area2007    19886 non-null  float64\n",
      " 14  Area2006    19536 non-null  float64\n",
      " 15  Area2005    19175 non-null  float64\n",
      " 16  Area2004    18846 non-null  float64\n",
      " 17  Area2003    18419 non-null  float64\n",
      " 18  Area2002    18042 non-null  float64\n",
      " 19  Area2001    17748 non-null  float64\n",
      " 20  Area2000    17287 non-null  float64\n",
      " 21  Area1999    16531 non-null  float64\n",
      "dtypes: float64(17), int64(5)\n",
      "memory usage: 4.0 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "BoundlessAreas = pd.read_csv(os.path.join(ResultsFolder, 'Areas1999to2015.csv'))\n",
    "print('Loaded Boundless dataset, whose settlements will be used as the index of the Fragmentation Index dataset. %s' \n",
    "      % time.ctime())\n",
    "print(BoundlessAreas.info())\n",
    "\n",
    "BoundedAreas = pd.read_csv(os.path.join(ResultsFolder, 'Areas1999to2015_Bounded.csv'))\n",
    "print('Loaded Bounded dataset, which will factor into the fragmentation calculation. %s' % time.ctime())\n",
    "print(BoundedAreas.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "261495a2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23490 entries, 0 to 23739\n",
      "Data columns (total 22 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Unnamed: 0  23490 non-null  int64  \n",
      " 1   Bounded_ID  23490 non-null  int64  \n",
      " 2   year        23490 non-null  int64  \n",
      " 3   ADM_ID      23490 non-null  int64  \n",
      " 4   Sett_ID     23490 non-null  int64  \n",
      " 5   Area2015    23490 non-null  float64\n",
      " 6   Area2014    22886 non-null  float64\n",
      " 7   Area2013    22456 non-null  float64\n",
      " 8   Area2012    21626 non-null  float64\n",
      " 9   Area2011    21315 non-null  float64\n",
      " 10  Area2010    20947 non-null  float64\n",
      " 11  Area2009    20469 non-null  float64\n",
      " 12  Area2008    20059 non-null  float64\n",
      " 13  Area2007    19708 non-null  float64\n",
      " 14  Area2006    19364 non-null  float64\n",
      " 15  Area2005    19005 non-null  float64\n",
      " 16  Area2004    18678 non-null  float64\n",
      " 17  Area2003    18254 non-null  float64\n",
      " 18  Area2002    17879 non-null  float64\n",
      " 19  Area2001    17587 non-null  float64\n",
      " 20  Area2000    17128 non-null  float64\n",
      " 21  Area1999    16379 non-null  float64\n",
      "dtypes: float64(17), int64(5)\n",
      "memory usage: 4.1 MB\n",
      "None\n",
      "Filtered the Bounded dataset to only rows where latest year's area is largest for each Sett_ID. Thu Jan 26 13:47:48 2023\n",
      "Renamed columns to avoid duplication during merge, and dropped unnecessary columns. Thu Jan 26 13:47:48 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23490 entries, 0 to 23489\n",
      "Data columns (total 40 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   Unnamed: 0_x  23490 non-null  int64  \n",
      " 1   Sett_ID       23490 non-null  int64  \n",
      " 2   year          23490 non-null  int64  \n",
      " 3   ADM_ID        23490 non-null  int64  \n",
      " 4   Area2015      23490 non-null  float64\n",
      " 5   Area2014      22886 non-null  float64\n",
      " 6   Area2013      22457 non-null  float64\n",
      " 7   Area2012      21629 non-null  float64\n",
      " 8   Area2011      21316 non-null  float64\n",
      " 9   Area2010      20948 non-null  float64\n",
      " 10  Area2009      20471 non-null  float64\n",
      " 11  Area2008      20062 non-null  float64\n",
      " 12  Area2007      19712 non-null  float64\n",
      " 13  Area2006      19369 non-null  float64\n",
      " 14  Area2005      19010 non-null  float64\n",
      " 15  Area2004      18683 non-null  float64\n",
      " 16  Area2003      18259 non-null  float64\n",
      " 17  Area2002      17884 non-null  float64\n",
      " 18  Area2001      17592 non-null  float64\n",
      " 19  Area2000      17134 non-null  float64\n",
      " 20  Area1999      16385 non-null  float64\n",
      " 21  Unnamed: 0_y  23490 non-null  int64  \n",
      " 22  Bounded_ID    23490 non-null  int64  \n",
      " 23  Largest2015   23490 non-null  float64\n",
      " 24  Largest2014   22886 non-null  float64\n",
      " 25  Largest2013   22456 non-null  float64\n",
      " 26  Largest2012   21626 non-null  float64\n",
      " 27  Largest2011   21315 non-null  float64\n",
      " 28  Largest2010   20947 non-null  float64\n",
      " 29  Largest2009   20469 non-null  float64\n",
      " 30  Largest2008   20059 non-null  float64\n",
      " 31  Largest2007   19708 non-null  float64\n",
      " 32  Largest2006   19364 non-null  float64\n",
      " 33  Largest2005   19005 non-null  float64\n",
      " 34  Largest2004   18678 non-null  float64\n",
      " 35  Largest2003   18254 non-null  float64\n",
      " 36  Largest2002   17879 non-null  float64\n",
      " 37  Largest2001   17587 non-null  float64\n",
      " 38  Largest2000   17128 non-null  float64\n",
      " 39  Largest1999   16379 non-null  float64\n",
      "dtypes: float64(34), int64(6)\n",
      "memory usage: 7.3 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "LargestFragments = BoundedAreas.loc[BoundedAreas.groupby([\"Sett_ID\"])[\"AREA2015\"].idxmax()] \n",
    "print(LargestFragments.info())\n",
    "print(\"Filtered the Bounded dataset to only rows where latest year's area is largest for each Sett_ID. %s\" % time.ctime())\n",
    "LargestFragments.columns = LargestFragments.columns.str.replace('AREA', 'Largest')\n",
    "LargestFragments = LargestFragments.drop(columns=['year', 'ADM_ID'])\n",
    "print(\"Renamed columns to avoid duplication during merge, and dropped unnecessary columns. %s\" % time.ctime())\n",
    "FragIndices = BoundlessAreas.merge(LargestFragments, how='left', on='Sett_ID')\n",
    "print(FragIndices.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "d7a828bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "del BoundlessAreas, BoundedAreas, LargestFragments"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "964d4c7b",
   "metadata": {},
   "source": [
    "### 7.2 Merge and run fragmentation calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "b17bcaeb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created names for Year 1999's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 1999. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2000's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2000. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2001's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2001. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2002's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2002. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2003's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2003. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2004's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2004. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2005's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2005. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2006's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2006. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2007's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2007. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2008's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2008. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2009's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2009. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2010's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2010. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2011's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2011. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2012's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2012. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2013's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2013. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2014's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2014. Thu Jan 26 13:47:48 2023\n",
      "Created names for Year 2015's variables and temporary objects. Thu Jan 26 13:47:48 2023\n",
      "Calculated fragmentation index for year 2015. Thu Jan 26 13:47:49 2023\n",
      "Completed fragmentation index calculations for all years. Thu Jan 26 13:47:49 2023\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 23490 entries, 0 to 23489\n",
      "Data columns (total 23 columns):\n",
      " #   Column        Non-Null Count  Dtype\n",
      "---  ------        --------------  -----\n",
      " 0   Unnamed: 0_x  23490 non-null  int64\n",
      " 1   Sett_ID       23490 non-null  int64\n",
      " 2   year          23490 non-null  int64\n",
      " 3   ADM_ID        23490 non-null  int64\n",
      " 4   Unnamed: 0_y  23490 non-null  int64\n",
      " 5   Bounded_ID    23490 non-null  int64\n",
      " 6   Frag1999      23490 non-null  int32\n",
      " 7   Frag2000      23490 non-null  int32\n",
      " 8   Frag2001      23490 non-null  int32\n",
      " 9   Frag2002      23490 non-null  int32\n",
      " 10  Frag2003      23490 non-null  int32\n",
      " 11  Frag2004      23490 non-null  int32\n",
      " 12  Frag2005      23490 non-null  int32\n",
      " 13  Frag2006      23490 non-null  int32\n",
      " 14  Frag2007      23490 non-null  int32\n",
      " 15  Frag2008      23490 non-null  int32\n",
      " 16  Frag2009      23490 non-null  int32\n",
      " 17  Frag2010      23490 non-null  int32\n",
      " 18  Frag2011      23490 non-null  int32\n",
      " 19  Frag2012      23490 non-null  int32\n",
      " 20  Frag2013      23490 non-null  int32\n",
      " 21  Frag2014      23490 non-null  int32\n",
      " 22  Frag2015      23490 non-null  int32\n",
      "dtypes: int32(17), int64(6)\n",
      "memory usage: 2.8 MB\n",
      "None\n",
      "       Unnamed: 0_x  Sett_ID  year  ADM_ID  Unnamed: 0_y  Bounded_ID  \\\n",
      "4202           4202    98249  2015      71          4229       98910   \n",
      "19045         19045   457584  2014     154         19266      462226   \n",
      "1927           1927    37252  2014     196          1936       37487   \n",
      "12828         12828   309863  2015     163         12980      313058   \n",
      "7029           7029   180129  2015     330          7079      181447   \n",
      "\n",
      "       Frag1999  Frag2000  Frag2001  Frag2002  ...  Frag2006  Frag2007  \\\n",
      "4202          0         0         0         0  ...         0         0   \n",
      "19045         0         0         0         0  ...         0         0   \n",
      "1927          0         0         0         0  ...         0         0   \n",
      "12828         0         0         0         0  ...         0         0   \n",
      "7029          0         0         0         0  ...         0         0   \n",
      "\n",
      "       Frag2008  Frag2009  Frag2010  Frag2011  Frag2012  Frag2013  Frag2014  \\\n",
      "4202          0         0         0         0         0         0         0   \n",
      "19045         0         0         0         0         0         0         0   \n",
      "1927          0         0         0         0         0         0         0   \n",
      "12828         0         0         0         0         0         0         0   \n",
      "7029          0         0         0         0         0         0         0   \n",
      "\n",
      "       Frag2015  \n",
      "4202          0  \n",
      "19045         0  \n",
      "1927          0  \n",
      "12828         0  \n",
      "7029          0  \n",
      "\n",
      "[5 rows x 23 columns]\n"
     ]
    }
   ],
   "source": [
    "for item in AllStudyYears:\n",
    "    YY = str(item) # 4-digit year\n",
    "    AreaYY = ''.join([\"AREA\", YY]) # The Boundless area variable name\n",
    "    LargestYY = ''.join(['Largest', YY]) # The Bounded largest area variable name\n",
    "    FragYY = ''.join([\"Frag\", YY]) # Name for the fragmentation index variable\n",
    "    print(\"Created names for Year %s's variables and temporary objects. %s\" % (item, time.ctime()))\n",
    "    \n",
    "    FragIndices[FragYY] = ((FragIndices[AreaYY] - FragIndices[LargestYY]) / FragIndices[AreaYY]) * 100\n",
    "    FragIndices[FragYY] = (FragIndices[FragYY].fillna(0).replace([np.inf, -np.inf], 0)).astype('int')\n",
    "    print(\"Calculated fragmentation index for year %s. %s\" % (item, time.ctime()))\n",
    "\n",
    "# Remove unnecessary columns.\n",
    "FragIndices = FragIndices.loc[:, ~FragIndices.columns.str.startswith('Largest')]\n",
    "FragIndices = FragIndices.loc[:, ~FragIndices.columns.str.startswith('AREA')]\n",
    "\n",
    "print('Completed fragmentation index calculations for all years. %s' % time.ctime())\n",
    "print(FragIndices.info())\n",
    "print(FragIndices.sample(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "5972dd80",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved to file. Thu Jan 26 13:47:49 2023\n"
     ]
    }
   ],
   "source": [
    "FragIndices = FragIndices.drop(columns=['Unnamed: 0_x', 'Unnamed: 0_y', 'year', 'ADM_ID'])\n",
    "FragIndices.to_csv(os.path.join(ResultsFolder, 'FragIndex%sto%s.csv' % (1999, 2015)))\n",
    "print('Saved to file. %s' % time.ctime())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "df90dce8",
   "metadata": {},
   "outputs": [],
   "source": [
    "del FragIndices"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec36f039",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "585db18c",
   "metadata": {},
   "source": [
    "## 8. PREPARE YEARLY DATASETS: POPULATION\n",
    "Can use this as a template for other annualized rasters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d8bd172",
   "metadata": {},
   "source": [
    "### 8.1 Reproject and reclassify with settlement buffer mask.\n",
    "Reclassify so that we only need to work with cells within X distance of settlements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "id": "205abf62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['bfa_ppp_2000_UNadj.tif', 'bfa_ppp_2001_UNadj.tif', 'bfa_ppp_2002_UNadj.tif', 'bfa_ppp_2003_UNadj.tif', 'bfa_ppp_2004_UNadj.tif', 'bfa_ppp_2005_UNadj.tif', 'bfa_ppp_2006_UNadj.tif', 'bfa_ppp_2007_UNadj.tif', 'bfa_ppp_2008_UNadj.tif', 'bfa_ppp_2009_UNadj.tif', 'bfa_ppp_2010_UNadj.tif', 'bfa_ppp_2011_UNadj.tif', 'bfa_ppp_2012_UNadj.tif', 'bfa_ppp_2013_UNadj.tif', 'bfa_ppp_2014_UNadj.tif', 'bfa_ppp_2015_UNadj.tif']\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:43:33 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2000_UNadj.tif. Sun Feb 19 10:43:36 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:43:38 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:43:38 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:43:41 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2001_UNadj.tif. Sun Feb 19 10:43:45 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:43:46 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:43:46 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:43:49 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2002_UNadj.tif. Sun Feb 19 10:43:53 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:43:54 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:43:54 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:43:57 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2003_UNadj.tif. Sun Feb 19 10:44:01 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:02 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:02 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:44:05 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2004_UNadj.tif. Sun Feb 19 10:44:08 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:10 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:10 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:44:13 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2005_UNadj.tif. Sun Feb 19 10:44:16 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:18 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:18 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:44:21 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2006_UNadj.tif. Sun Feb 19 10:44:24 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:26 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:26 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:44:29 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2007_UNadj.tif. Sun Feb 19 10:44:32 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:34 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:34 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:44:37 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2008_UNadj.tif. Sun Feb 19 10:44:41 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:42 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:42 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:44:46 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2009_UNadj.tif. Sun Feb 19 10:44:49 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:51 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:51 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:44:54 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2010_UNadj.tif. Sun Feb 19 10:44:57 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:44:59 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:44:59 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:45:02 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2011_UNadj.tif. Sun Feb 19 10:45:05 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:45:07 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:45:07 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:45:10 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2012_UNadj.tif. Sun Feb 19 10:45:13 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:45:15 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:45:15 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:45:18 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2013_UNadj.tif. Sun Feb 19 10:45:22 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:45:23 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:45:23 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:45:27 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2014_UNadj.tif. Sun Feb 19 10:45:30 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:45:31 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:45:31 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:45:35 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for bfa_ppp_2015_UNadj.tif. Sun Feb 19 10:45:38 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:45:39 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:45:39 2023 \n",
      "\n",
      "\n",
      " \n",
      " Finished all years in list. Sun Feb 19 10:45:39 2023\n"
     ]
    }
   ],
   "source": [
    "MaskByZone(MaskPath=r'Results/Catchment.gpkg', MaskLayerName = \"Buff2000m_2015\", \n",
    "           SourceFolder='Population/SourceFiles', DestFolder='Population')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "id": "aa0f89e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Msk_2000.tif', 'Msk_2001.tif', 'Msk_2002.tif', 'Msk_2003.tif', 'Msk_2004.tif', 'Msk_2005.tif', 'Msk_2006.tif', 'Msk_2007.tif', 'Msk_2008.tif', 'Msk_2009.tif', 'Msk_2010.tif', 'Msk_2011.tif', 'Msk_2012.tif', 'Msk_2013.tif', 'Msk_2014.tif', 'Msk_2015.tif', 'PreDefinedFunction', 'SourceFiles']\n"
     ]
    }
   ],
   "source": [
    "print(os.listdir('Population/'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f624d801",
   "metadata": {},
   "source": [
    "### 8.2 Raster values summarized by settlement.\n",
    "1. Convert each annualized raster to .xyz, \n",
    "2. then bring them to vector space and assign their Sett_ID,\n",
    "3. and finally, aggregate the value as appropriate to the settlement level and save table to file.\n",
    "\n",
    "XYZ is similar to .csv. Raster cell centers are stored as x and y, and their value is stored as z."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "id": "8801654c",
   "metadata": {},
   "outputs": [],
   "source": [
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS_equalarea')[['Sett_ID', 'geometry']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "id": "1de7230c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Msk_2000.tif', 'Msk_2001.tif', 'Msk_2002.tif', 'Msk_2003.tif', 'Msk_2004.tif', 'Msk_2005.tif', 'Msk_2006.tif', 'Msk_2007.tif', 'Msk_2008.tif', 'Msk_2009.tif', 'Msk_2010.tif', 'Msk_2011.tif', 'Msk_2012.tif', 'Msk_2013.tif', 'Msk_2014.tif', 'Msk_2015.tif']\n",
      "       Sett_ID\n",
      "0            1\n",
      "1            2\n",
      "2            3\n",
      "3            4\n",
      "4            5\n",
      "...        ...\n",
      "23485   612616\n",
      "23486   612723\n",
      "23487   612934\n",
      "23488   612944\n",
      "23489   615066\n",
      "\n",
      "[23490 rows x 1 columns]\n",
      "Loading data for Msk_2000.tif. Sun Feb 19 10:46:44 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2000. Sun Feb 19 10:48:15 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:49:59 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:50:03 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 11:01:22 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "55887160  0.219118      794\n",
      "51293238  0.310020    30101\n",
      "29133546  0.168024   128526\n",
      "45048797  0.131523    41163\n",
      "28237162  0.305185   388601\n",
      "54696242  0.305337    93183\n",
      "22595607  0.484282   409877\n",
      "45255061  0.131138    41100\n",
      "36659200  0.588957   359349\n",
      "13499361  0.125563   444644\n",
      "\n",
      "Exported as table. Sun Feb 19 11:02:05 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 11:02:05 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2000. Sun Feb 19 11:02:06 2023 \n",
      "\n",
      "       Sett_ID  POPct2000   POPsum2000\n",
      "1507     35683      580.0   148.609655\n",
      "19691   458371      341.0   394.664651\n",
      "22161   525266      568.0   235.284028\n",
      "3478     84391     1512.0   416.471300\n",
      "17035   434839     1000.0   375.101288\n",
      "22678   568642     1815.0   168.007929\n",
      "1349     30190     1156.0   246.842404\n",
      "11493   265984      611.0   410.953914\n",
      "5266    129732     1405.0  1696.180766\n",
      "1201     30000     1108.0   203.613947\n",
      "Loading data for Msk_2001.tif. Sun Feb 19 11:02:06 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2001. Sun Feb 19 11:03:39 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 11:05:19 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 11:05:23 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 11:16:46 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "40579210  0.096367   351321\n",
      "36026057  0.971124   180086\n",
      "39574830  0.338412   358663\n",
      "44759368  0.470439   351272\n",
      "54390159  0.071363      575\n",
      "44144246  0.312747   103818\n",
      "47390391  0.748308    36855\n",
      "46087246  0.280388   351238\n",
      "24616065  0.292199   508995\n",
      "25286760  0.579077   409212\n",
      "\n",
      "Exported as table. Sun Feb 19 11:17:29 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 11:17:30 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2001. Sun Feb 19 11:17:30 2023 \n",
      "\n",
      "       Sett_ID  POPct2000   POPsum2000  POPct2001   POPsum2001\n",
      "79          95     1553.0   128.825846     1553.0   149.983500\n",
      "7404    180600     1119.0   871.623840     1119.0   979.967499\n",
      "117        145      868.0   259.081650      868.0   298.518177\n",
      "14703   400532     1070.0   438.913321     1070.0   529.568346\n",
      "18670   457093      332.0   150.011172      332.0   135.374446\n",
      "22835   568955     1206.0   172.326601     1206.0   157.055014\n",
      "10167   227458     1077.0   332.479935     1077.0   317.496592\n",
      "22388   528449     1061.0  1213.926921     1061.0  1229.964921\n",
      "21024   475085      803.0   258.912247      803.0   239.280986\n",
      "6612    165779     1548.0   163.421024     1548.0   141.666932\n",
      "Loading data for Msk_2002.tif. Sun Feb 19 11:17:30 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2002. Sun Feb 19 11:19:04 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 11:20:46 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 11:20:49 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 11:32:31 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "43030842  0.339980    51172\n",
      "17552532  0.544799   456993\n",
      "44476990  0.550600   156090\n",
      "25547111  0.738206   409179\n",
      "14405089  0.228106   474567\n",
      "47749231  0.239537    36648\n",
      "33901107  0.744797   215818\n",
      "49205187  0.144390   100370\n",
      "28596924  0.231378   397758\n",
      "23149298  0.487783   393657\n",
      "\n",
      "Exported as table. Sun Feb 19 11:33:16 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 11:33:17 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2002. Sun Feb 19 11:33:17 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "12074   277909      916.0  672.226508      916.0  823.010407      916.0   \n",
      "14544   396586     2240.0  850.826942     2240.0  884.264444     2240.0   \n",
      "5884    138174      837.0  324.830181      837.0  333.396291      837.0   \n",
      "2140     49358      868.0   80.343309      868.0   92.890319      868.0   \n",
      "8558    203763      123.0  224.862491      123.0  250.986340      123.0   \n",
      "8391    203493      894.0  327.105168      894.0  275.305898      894.0   \n",
      "13831   359245      898.0  119.555188      898.0   93.809884      898.0   \n",
      "3059     65461       43.0  136.172075       43.0  164.127749       43.0   \n",
      "2483     51621     1378.0  330.099066     1378.0  357.844221     1378.0   \n",
      "6371    157605      171.0   95.472575      171.0   83.833581      171.0   \n",
      "\n",
      "       POPsum2002  \n",
      "12074  807.199599  \n",
      "14544  854.937752  \n",
      "5884   326.825070  \n",
      "2140    98.893365  \n",
      "8558   243.785296  \n",
      "8391   283.698782  \n",
      "13831   99.176099  \n",
      "3059   159.055165  \n",
      "2483   371.768575  \n",
      "6371    93.180949  \n",
      "Loading data for Msk_2003.tif. Sun Feb 19 11:33:18 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2003. Sun Feb 19 11:34:52 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 11:36:35 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 11:36:40 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 11:48:20 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "19365880  0.454144   447826\n",
      "26093782  0.307960   396550\n",
      "40542035  0.627519   277738\n",
      "31839291  0.452701   319102\n",
      "26353040  0.166452   388679\n",
      "39805755  0.472744   247719\n",
      "7246929   0.316157   581615\n",
      "31730640  1.200685   294620\n",
      "41177029  0.098436   188429\n",
      "16583825  1.777928   458138\n",
      "\n",
      "Exported as table. Sun Feb 19 11:49:05 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 11:49:06 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2003. Sun Feb 19 11:49:06 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "13511   319171     1265.0  136.329117     1265.0  107.085618     1265.0   \n",
      "9386    215240      287.0   67.238044      287.0   62.992048      287.0   \n",
      "6200    156011      837.0  283.754890      837.0  318.640794      837.0   \n",
      "19559   458209      330.0   42.254054      330.0   38.849343      330.0   \n",
      "20504   474194      111.0   29.255816      111.0   31.361572      111.0   \n",
      "19381   457986      338.0  188.083088      338.0  150.987301      338.0   \n",
      "208        246      126.0   49.925390      126.0   47.514428      126.0   \n",
      "17059   438414      473.0  105.549629      473.0  109.375221      473.0   \n",
      "4424    102668      288.0   98.984170      288.0   94.618274      288.0   \n",
      "7362    180546      149.0   90.876237      149.0   89.658701      149.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  \n",
      "13511  117.942442     1265.0  128.609941  \n",
      "9386    67.157056      287.0   62.846289  \n",
      "6200   311.607378      837.0  343.821694  \n",
      "19559   38.953072      330.0   49.322209  \n",
      "20504   32.724952      111.0   34.502791  \n",
      "19381  181.150317      338.0  195.913201  \n",
      "208     55.747243      126.0   55.638743  \n",
      "17059  113.276905      473.0  116.636217  \n",
      "4424   106.732533      288.0  112.940132  \n",
      "7362    89.350756      149.0   85.250950  \n",
      "Loading data for Msk_2004.tif. Sun Feb 19 11:49:07 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2004. Sun Feb 19 11:50:42 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 11:52:23 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 11:52:27 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 12:03:59 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "40554760  0.450365    51769\n",
      "11013109  3.073218   474186\n",
      "59064242  0.059327     5560\n",
      "41267226  0.274766   267182\n",
      "21894497  0.453424   400487\n",
      "20291868  0.307123   528441\n",
      "30616075  0.767993   203959\n",
      "42349521  0.316110    73686\n",
      "16604284  0.471777   580910\n",
      "45300689  0.196453    51310\n",
      "\n",
      "Exported as table. Sun Feb 19 12:04:45 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 12:04:45 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2004. Sun Feb 19 12:04:46 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "8136    203083      229.0  273.250792      229.0  254.509717      229.0   \n",
      "12663   309169      569.0  416.465129      569.0  377.242768      569.0   \n",
      "12890   309946      524.0  383.718624      524.0  385.676175      524.0   \n",
      "21978   509488      636.0  255.886882      636.0  255.608182      636.0   \n",
      "16910   417499      396.0  112.057415      396.0  118.366495      396.0   \n",
      "3467     84363     1201.0  181.312967     1201.0  181.806501     1201.0   \n",
      "14343   395798       58.0   20.637580       58.0   19.837348       58.0   \n",
      "21646   505532       98.0   83.697168       98.0   83.178906       98.0   \n",
      "42          46      723.0   87.574351      723.0  126.391838      723.0   \n",
      "4352    102563      638.0  112.448986      638.0  117.734817      638.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  POPsum2004  \n",
      "8136   269.308663      229.0  284.961560      229.0  288.617111  \n",
      "12663  394.919186      569.0  390.109625      569.0  387.359721  \n",
      "12890  396.615189      524.0  433.184482      524.0  432.164173  \n",
      "21978  266.456560      636.0  262.117983      636.0  271.611845  \n",
      "16910  132.542465      396.0  135.655614      396.0  134.890430  \n",
      "3467   193.591270     1201.0  203.298146     1201.0  205.517782  \n",
      "14343   18.480633       58.0   19.097686       58.0   19.196028  \n",
      "21646   82.100271       98.0   83.597938       98.0   83.054143  \n",
      "42     126.922238      723.0  136.389024      723.0  137.621560  \n",
      "4352   117.406197      638.0  122.038492      638.0  130.530720  \n",
      "Loading data for Msk_2005.tif. Sun Feb 19 12:04:46 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2005. Sun Feb 19 12:06:22 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 12:08:07 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 12:08:11 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 12:19:22 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "32026672  1.474894   302886\n",
      "18020555  0.145767   445892\n",
      "54253968  0.379169    29872\n",
      "25493904  0.391546   408535\n",
      "21884553  0.398975   388531\n",
      "28525049  0.538647   396418\n",
      "54235873  0.581846    29899\n",
      "10907317  0.480491   581549\n",
      "59126673  0.169112     5652\n",
      "28686513  0.505155   397730\n",
      "\n",
      "Exported as table. Sun Feb 19 12:20:06 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 12:20:06 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2005. Sun Feb 19 12:20:07 2023 \n",
      "\n",
      "       Sett_ID  POPct2000   POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "1349     30190     1156.0   246.842404     1156.0  258.485434     1156.0   \n",
      "16697   417039      946.0   278.420957      946.0  260.766323      946.0   \n",
      "14086   388727     2092.0   157.304067     2092.0  162.725319     2092.0   \n",
      "5668    137851      199.0    42.382645      199.0   41.429290      199.0   \n",
      "9591    215699     2285.0   622.032239     2285.0  555.810591     2285.0   \n",
      "5558    137704     1554.0   516.556262     1554.0  654.327331     1554.0   \n",
      "2280     51329      754.0  1017.872881      754.0  996.011343      754.0   \n",
      "17168   445209     1284.0   414.130127     1284.0  414.047675     1284.0   \n",
      "7532    183689      555.0   159.709173      555.0  153.145202      555.0   \n",
      "14042   388681      992.0   235.886053      992.0  272.942078      992.0   \n",
      "\n",
      "        POPsum2002  POPct2003   POPsum2003  POPct2004   POPsum2004  POPct2005  \\\n",
      "1349    258.503141     1156.0   261.057993     1156.0   270.652965     1156.0   \n",
      "16697   275.566852      946.0   275.777021      946.0   287.141320      946.0   \n",
      "14086   171.072311     2092.0   182.998864     2092.0   168.951820     2092.0   \n",
      "5668     42.575624      199.0    41.762550      199.0    71.292730      199.0   \n",
      "9591    564.219539     2285.0   558.906871     2285.0   565.228297     2285.0   \n",
      "5558    697.679776     1554.0   699.795713     1554.0   709.965290     1554.0   \n",
      "2280   1020.195576      754.0  1017.798141      754.0  1042.830173      754.0   \n",
      "17168   519.067046     1284.0   542.447280     1284.0   520.681810     1284.0   \n",
      "7532    157.413220      555.0   162.738950      555.0   177.178891      555.0   \n",
      "14042   253.427491      992.0   256.203901      992.0   253.828752      992.0   \n",
      "\n",
      "        POPsum2005  \n",
      "1349    285.194610  \n",
      "16697   289.794421  \n",
      "14086   184.978857  \n",
      "5668     78.402773  \n",
      "9591    576.363592  \n",
      "5558    783.079025  \n",
      "2280   1052.507049  \n",
      "17168   555.460594  \n",
      "7532    173.822577  \n",
      "14042   290.068079  \n",
      "Loading data for Msk_2006.tif. Sun Feb 19 12:20:07 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2006. Sun Feb 19 12:21:40 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 12:23:21 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 12:23:25 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 12:34:36 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "25456833  0.280067   408859\n",
      "16457678  0.106185   457055\n",
      "53070313  0.421626      224\n",
      "33965183  0.529891   309787\n",
      "18433456  0.509521   417005\n",
      "15641967  0.843566   456909\n",
      "24019678  0.327545   392967\n",
      "13426061  0.380586   468440\n",
      "37659783  0.277279   198967\n",
      "30113517  0.659997   180603\n",
      "\n",
      "Exported as table. Sun Feb 19 12:35:19 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 12:35:20 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2006. Sun Feb 19 12:35:20 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "21240   494882      918.0  215.338280      918.0  212.702940      918.0   \n",
      "23253   581625      390.0   63.620949      390.0   54.590880      390.0   \n",
      "8527    203716      118.0  153.467988      118.0  148.453276      118.0   \n",
      "16317   416093      493.0  524.920873      493.0  574.819306      493.0   \n",
      "8737    204075       88.0   77.166734       88.0   78.059886       88.0   \n",
      "11655   270740      324.0  169.195409      324.0  157.790254      324.0   \n",
      "15964   411860      453.0  217.328324      453.0  208.483284      453.0   \n",
      "23269   583959      671.0  126.285158      671.0  121.900771      671.0   \n",
      "15863   409834      300.0  199.168976      300.0  232.720965      300.0   \n",
      "9369    215206      826.0  282.541097      826.0  276.660318      826.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  POPsum2004  POPct2005  \\\n",
      "21240  212.371600      918.0  215.189416      918.0  217.489958      918.0   \n",
      "23253   59.361656      390.0   64.741606      390.0   65.995190      390.0   \n",
      "8527   153.530040      118.0  157.370301      118.0  148.260360      118.0   \n",
      "16317  541.538200      493.0  514.888246      493.0  568.840427      493.0   \n",
      "8737    78.258132       88.0   85.884363       88.0   79.880367       88.0   \n",
      "11655  155.917075      324.0  154.164802      324.0  157.034240      324.0   \n",
      "15964  210.119131      453.0  210.708791      453.0  212.784458      453.0   \n",
      "23269  128.815818      671.0  129.004180      671.0  125.073638      671.0   \n",
      "15863  246.262669      300.0  243.856027      300.0  261.101877      300.0   \n",
      "9369   282.695683      826.0  289.986772      826.0  285.140359      826.0   \n",
      "\n",
      "       POPsum2005  POPct2006  POPsum2006  \n",
      "21240  219.054800      918.0  223.836112  \n",
      "23253  111.673280      390.0  108.881172  \n",
      "8527   150.715513      118.0  151.456452  \n",
      "16317  574.074777      493.0  580.279007  \n",
      "8737    88.050769       88.0   92.285607  \n",
      "11655  152.132695      324.0  154.804059  \n",
      "15964  215.548645      453.0  215.180785  \n",
      "23269  137.189214      671.0  134.132477  \n",
      "15863  257.506028      300.0  272.001498  \n",
      "9369   282.808594      826.0  284.716674  \n",
      "Loading data for Msk_2007.tif. Sun Feb 19 12:35:21 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2007. Sun Feb 19 12:36:53 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 12:38:34 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 12:38:38 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 12:49:50 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "54919303  0.087183      497\n",
      "36871915  1.569703   248026\n",
      "10868765  0.290372   468599\n",
      "27135385  0.795659   409379\n",
      "18459470  1.650253   456902\n",
      "38513747  0.793987   247808\n",
      "24433372  0.167170   396625\n",
      "18658894  0.164547   528749\n",
      "30803924  0.201081   396634\n",
      "8895800   0.089933   474502\n",
      "\n",
      "Exported as table. Sun Feb 19 12:50:34 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 12:50:34 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2007. Sun Feb 19 12:50:35 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "6491    161862      436.0  213.438821      436.0  208.338779      436.0   \n",
      "17054   438138      455.0  114.211714      455.0  115.707912      455.0   \n",
      "20890   474822       47.0   11.440905       47.0   11.774504       47.0   \n",
      "11612   267099      970.0  387.720869      970.0  403.965692      970.0   \n",
      "23278   586115      758.0  161.871429      758.0  191.956104      758.0   \n",
      "23260   583210      415.0  184.081740      415.0  174.158159      415.0   \n",
      "18887   457384      383.0  114.446471      383.0  111.680996      383.0   \n",
      "5838    138108      259.0   79.220646      259.0   77.682206      259.0   \n",
      "9812    220887      524.0  314.134162      524.0  312.666766      524.0   \n",
      "7582    185243      280.0  131.412057      280.0  136.251061      280.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  POPsum2004  POPct2005  \\\n",
      "6491   222.896406      436.0  230.196118      436.0  224.294909      436.0   \n",
      "17054  122.048005      455.0  126.395379      455.0  130.030776      455.0   \n",
      "20890   12.090019       47.0   12.194325       47.0   11.798912       47.0   \n",
      "11612  415.327727      970.0  424.268723      970.0  424.525986      970.0   \n",
      "23278  209.296885      758.0  213.459026      758.0  217.911139      758.0   \n",
      "23260  181.039933      415.0  180.690671      415.0  191.137846      415.0   \n",
      "18887  123.078186      383.0  143.998239      383.0  143.318282      383.0   \n",
      "5838    78.341598      259.0   82.035501      259.0   79.160249      259.0   \n",
      "9812   284.496139      524.0  274.078072      524.0  291.908838      524.0   \n",
      "7582   139.985441      280.0  148.677836      280.0  146.962314      280.0   \n",
      "\n",
      "       POPsum2005  POPct2006  POPsum2006  POPct2007  POPsum2007  \n",
      "6491   227.867548      436.0  229.142878      436.0  231.823231  \n",
      "17054  129.961029      455.0  135.584857      455.0  140.519850  \n",
      "20890   12.588668       47.0   12.404459       47.0   18.714439  \n",
      "11612  432.131623      970.0  441.822368      970.0  455.540206  \n",
      "23278  230.752971      758.0  236.544407      758.0  246.429315  \n",
      "23260  201.435144      415.0  214.057952      415.0  216.594254  \n",
      "18887  147.287214      383.0  146.713318      383.0  151.380592  \n",
      "5838    83.947158      259.0   82.121639      259.0   84.226150  \n",
      "9812   317.906815      524.0  295.183023      524.0  299.094651  \n",
      "7582   147.944282      280.0  145.305675      280.0  149.523692  \n",
      "Loading data for Msk_2008.tif. Sun Feb 19 12:50:35 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2008. Sun Feb 19 12:52:08 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 12:53:49 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 12:53:53 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 13:05:04 2023 \n",
      "\n",
      "                  Z  Sett_ID\n",
      "36346694   0.737197    79380\n",
      "18691326   0.382482   400799\n",
      "46022757   1.125914   236590\n",
      "59476911   0.131678       74\n",
      "38682511   0.413280   188656\n",
      "23611174   1.666904   509259\n",
      "27846967   0.201207   556874\n",
      "33207889   0.344652   137959\n",
      "31325949  26.140343   227047\n",
      "9200335    0.204689   474496\n",
      "\n",
      "Exported as table. Sun Feb 19 13:05:47 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 13:05:48 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2008. Sun Feb 19 13:05:48 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "7833    188493     1146.0  273.015936     1146.0  298.224525     1146.0   \n",
      "15739   409670      240.0  170.207903      240.0  163.305714      240.0   \n",
      "787      14555      638.0  251.925123      638.0  209.341772      638.0   \n",
      "167        200     1636.0  294.951368     1636.0  322.204570     1636.0   \n",
      "16830   417325      487.0  285.918188      487.0  302.966940      487.0   \n",
      "10024   227271       84.0  168.056486       84.0  173.534096       84.0   \n",
      "18279   454263      166.0  155.202309      166.0  149.425771      166.0   \n",
      "518        662      843.0  174.214897      843.0  163.526390      843.0   \n",
      "21686   508697     1481.0  786.728673     1481.0  913.601066     1481.0   \n",
      "3773     87503     1282.0  168.715842     1282.0  253.798685     1282.0   \n",
      "\n",
      "       POPsum2002  POPct2003   POPsum2003  POPct2004   POPsum2004  POPct2005  \\\n",
      "7833   287.822225     1146.0   294.565467     1146.0   289.211298     1146.0   \n",
      "15739  177.979714      240.0   181.189212      240.0   189.073976      240.0   \n",
      "787    260.273314      638.0   264.716534      638.0   254.473641      638.0   \n",
      "167    337.542154     1636.0   508.034576     1636.0   505.625973     1636.0   \n",
      "16830  304.800917      487.0   305.474922      487.0   329.464192      487.0   \n",
      "10024  167.988083       84.0   155.432285       84.0   171.811314       84.0   \n",
      "18279  151.182280      166.0   151.990963      166.0   153.973191      166.0   \n",
      "518    177.438241      843.0   163.507567      843.0   185.815102      843.0   \n",
      "21686  969.526737     1481.0  1014.610534     1481.0  1091.622545     1481.0   \n",
      "3773   239.917847     1282.0   283.493612     1282.0   279.148007     1282.0   \n",
      "\n",
      "        POPsum2005  POPct2006   POPsum2006  POPct2007   POPsum2007  POPct2008  \\\n",
      "7833    282.881007     1146.0   285.260319     1146.0   289.826741     1146.0   \n",
      "15739   197.679514      240.0   187.706462      240.0   203.885610      240.0   \n",
      "787     264.339175      638.0   262.440243      638.0   269.842499      638.0   \n",
      "167     498.395771     1636.0   545.206061     1636.0   578.409156     1636.0   \n",
      "16830   321.634381      487.0   323.520126      487.0   337.577807      487.0   \n",
      "10024   161.005727       84.0   155.531381       84.0   163.164509       84.0   \n",
      "18279   162.266113      166.0   159.902535      166.0   166.933646      166.0   \n",
      "518     214.528241      843.0   231.176691      843.0   297.997488      843.0   \n",
      "21686  1203.331265     1481.0  1192.221412     1481.0  1116.850461     1481.0   \n",
      "3773    304.117863     1282.0   279.086035     1282.0   294.699743     1282.0   \n",
      "\n",
      "        POPsum2008  \n",
      "7833    297.669679  \n",
      "15739   198.456441  \n",
      "787     295.530374  \n",
      "167     607.132079  \n",
      "16830   334.324069  \n",
      "10024   167.092463  \n",
      "18279   167.806376  \n",
      "518     287.238373  \n",
      "21686  1242.628539  \n",
      "3773    298.136921  \n",
      "Loading data for Msk_2009.tif. Sun Feb 19 13:05:49 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2009. Sun Feb 19 13:07:22 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 13:09:03 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 13:09:07 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 13:20:18 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "37773036  0.343987    37184\n",
      "32457098  0.483514   294439\n",
      "51941329  0.203162    87501\n",
      "19849342  0.269860   400754\n",
      "33487226  1.076863   203240\n",
      "52856936  0.405903    87459\n",
      "16135031  0.261119   457068\n",
      "31934165  0.448403   137702\n",
      "32547808  0.286394   318876\n",
      "37690818  0.431190   358776\n",
      "\n",
      "Exported as table. Sun Feb 19 13:21:01 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 13:21:01 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2009. Sun Feb 19 13:21:02 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "15427   409208      677.0  591.311011      677.0  697.414352      677.0   \n",
      "9684    215873      767.0  522.469688      767.0  553.245601      767.0   \n",
      "725      12535      301.0   67.843032      301.0   67.858639      301.0   \n",
      "4599    110686      124.0   28.120471      124.0   26.675285      124.0   \n",
      "3783     87518      751.0  511.363245      751.0  596.749009      751.0   \n",
      "1400     30630      240.0   91.588288      240.0   90.854656      240.0   \n",
      "2178     51197     1037.0  573.876984     1037.0  543.001012     1037.0   \n",
      "5524    136727     1125.0  327.619621     1125.0  315.134465     1125.0   \n",
      "11157   248078      651.0  445.092189      651.0  456.557639      651.0   \n",
      "13543   325230       59.0   43.657336       59.0   40.235109       59.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  ...  POPct2005  \\\n",
      "15427  686.489101      677.0  695.937403      677.0  ...      677.0   \n",
      "9684   591.133009      767.0  578.171783      767.0  ...      767.0   \n",
      "725     71.809972      301.0   83.108851      301.0  ...      301.0   \n",
      "4599    28.364470      124.0   27.141448      124.0  ...      124.0   \n",
      "3783   622.890516      751.0  580.422809      751.0  ...      751.0   \n",
      "1400    93.969767      240.0   92.925573      240.0  ...      240.0   \n",
      "2178   521.883066     1037.0  511.187541     1037.0  ...     1037.0   \n",
      "5524   303.645252     1125.0  308.075855     1125.0  ...     1125.0   \n",
      "11157  461.146878      651.0  456.991575      651.0  ...      651.0   \n",
      "13543   39.076112       59.0   38.965017       59.0  ...       59.0   \n",
      "\n",
      "       POPsum2005  POPct2006  POPsum2006  POPct2007  POPsum2007  POPct2008  \\\n",
      "15427  711.485077      677.0  763.761305      677.0  747.132743      677.0   \n",
      "9684   596.692715      767.0  581.680956      767.0  616.012502      767.0   \n",
      "725     87.652128      301.0   92.354240      301.0   99.255644      301.0   \n",
      "4599    32.451738      124.0   43.327578      124.0   43.793857      124.0   \n",
      "3783   633.188567      751.0  651.928948      751.0  643.618552      751.0   \n",
      "1400   100.791914      240.0  102.201869      240.0  110.276990      240.0   \n",
      "2178   643.498250     1037.0  677.371394     1037.0  675.692798     1037.0   \n",
      "5524   319.278736     1125.0  330.430341     1125.0  333.952215     1125.0   \n",
      "11157  480.810021      651.0  483.598441      651.0  501.502366      651.0   \n",
      "13543   60.903197       59.0   62.710866       59.0   61.270017       59.0   \n",
      "\n",
      "       POPsum2008  POPct2009  POPsum2009  \n",
      "15427  757.591165      677.0  768.790476  \n",
      "9684   611.292036      767.0  613.441231  \n",
      "725    103.290130      301.0  105.944377  \n",
      "4599    45.868308      124.0   44.046134  \n",
      "3783   616.975258      751.0  655.616823  \n",
      "1400   110.866638      240.0  115.163635  \n",
      "2178   722.527338     1037.0  740.269120  \n",
      "5524   344.596278     1125.0  358.238537  \n",
      "11157  516.862276      651.0  520.645676  \n",
      "13543   66.085748       59.0   65.488778  \n",
      "\n",
      "[10 rows x 21 columns]\n",
      "Loading data for Msk_2010.tif. Sun Feb 19 13:21:03 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2010. Sun Feb 19 13:22:37 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 13:24:18 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 13:24:22 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 13:35:33 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "34177246  0.381088   137691\n",
      "13481458  0.190855   417494\n",
      "39192046  0.362939   102546\n",
      "52307184  0.385410    30030\n",
      "30857431  0.155778   138120\n",
      "34543809  0.681012   117355\n",
      "27056853  0.710978   534834\n",
      "23805876  0.246505   400530\n",
      "43022592  0.404532    51778\n",
      "40332419  0.390501   129834\n",
      "\n",
      "Exported as table. Sun Feb 19 13:36:17 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 13:36:17 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2010. Sun Feb 19 13:36:18 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "3835     90544      614.0  109.335143      614.0  109.073210      614.0   \n",
      "94         120      887.0   22.636840      887.0   24.891955      887.0   \n",
      "10489   235418       12.0  265.505316       12.0  280.845341       12.0   \n",
      "4193     98238      678.0  203.422169      678.0  233.196048      678.0   \n",
      "2608     51778     1869.0  477.721760     1869.0  563.690699     1869.0   \n",
      "7772    188384     2167.0  219.008251     2167.0  361.052577     2167.0   \n",
      "11116   248002      360.0  285.793193      360.0  298.927586      360.0   \n",
      "14931   400830      726.0   72.590051      726.0   70.605373      726.0   \n",
      "16017   414004      342.0  293.587580      342.0  297.966914      342.0   \n",
      "21562   502072      525.0  225.262453      525.0  215.321556      525.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  ...  POPct2006  \\\n",
      "3835   124.490765      614.0  125.958345      614.0  ...      614.0   \n",
      "94      32.028062      887.0   32.024463      887.0  ...      887.0   \n",
      "10489  295.479078       12.0  318.032368       12.0  ...       12.0   \n",
      "4193   227.588614      678.0  222.412406      678.0  ...      678.0   \n",
      "2608   540.810701     1869.0  582.088021     1869.0  ...     1869.0   \n",
      "7772   383.491587     2167.0  416.333568     2167.0  ...     2167.0   \n",
      "11116  302.087168      360.0  303.348715      360.0  ...      360.0   \n",
      "14931   72.672765      726.0   75.672026      726.0  ...      726.0   \n",
      "16017  306.799168      342.0  294.219838      342.0  ...      342.0   \n",
      "21562  239.044569      525.0  241.896505      525.0  ...      525.0   \n",
      "\n",
      "       POPsum2006  POPct2007  POPsum2007  POPct2008  POPsum2008  POPct2009  \\\n",
      "3835   138.290851      614.0  141.706131      614.0  153.144536      614.0   \n",
      "94      48.498751      887.0   53.792814      887.0   65.573684      887.0   \n",
      "10489  407.636929       12.0  392.529596       12.0  457.908199       12.0   \n",
      "4193   229.530412      678.0  230.812412      678.0  228.245077      678.0   \n",
      "2608   608.973133     1869.0  601.930638     1869.0  630.713045     1869.0   \n",
      "7772   457.221636     2167.0  496.321397     2167.0  524.684879     2167.0   \n",
      "11116  320.685124      360.0  328.224822      360.0  335.630907      360.0   \n",
      "14931   77.742881      726.0   82.562384      726.0   82.691220      726.0   \n",
      "16017  332.862118      342.0  339.494590      342.0  347.655319      342.0   \n",
      "21562  242.024686      525.0  253.450930      525.0  262.704472      525.0   \n",
      "\n",
      "       POPsum2009  POPct2010  POPsum2010  \n",
      "3835   157.185893      614.0  163.435330  \n",
      "94      98.848745      887.0  115.995254  \n",
      "10489  478.807056       12.0  510.052578  \n",
      "4193   230.105513      678.0  244.425403  \n",
      "2608   638.477310     1869.0  659.197373  \n",
      "7772   518.212624     2167.0  584.215798  \n",
      "11116  336.321410      360.0  358.588614  \n",
      "14931   87.073614      726.0   87.902924  \n",
      "16017  363.369877      342.0  366.502590  \n",
      "21562  253.399084      525.0  259.122921  \n",
      "\n",
      "[10 rows x 23 columns]\n",
      "Loading data for Msk_2011.tif. Sun Feb 19 13:36:18 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2011. Sun Feb 19 13:37:53 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 13:39:33 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 13:39:37 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 13:50:50 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "21028473  0.295534   574497\n",
      "33719238  0.264908   147689\n",
      "12323316  0.239984   474761\n",
      "19064220  0.390589   568807\n",
      "31775314  0.572395   303276\n",
      "23993861  0.462469   400514\n",
      "46352566  0.610375   156000\n",
      "13095372  0.278929   474803\n",
      "31226267  1.211829   203842\n",
      "32852727  1.038081   313793\n",
      "\n",
      "Exported as table. Sun Feb 19 13:51:34 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 13:51:34 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2011. Sun Feb 19 13:51:35 2023 \n",
      "\n",
      "       Sett_ID  POPct2000   POPsum2000  POPct2001   POPsum2001  POPct2002  \\\n",
      "10516   236284      581.0   253.178854      581.0   247.692178      581.0   \n",
      "19503   458144     1020.0  1333.303449     1020.0  1498.613779     1020.0   \n",
      "19991   462145      178.0    97.960341      178.0    89.278833      178.0   \n",
      "7741    187796     1210.0   411.573254     1210.0   455.651706     1210.0   \n",
      "16823   417316      290.0   327.960194      290.0   450.581712      290.0   \n",
      "19626   458295     1099.0   200.793978     1099.0   225.766738     1099.0   \n",
      "8238    203246      639.0   372.620069      639.0   369.979958      639.0   \n",
      "16857   417390     1608.0   481.758771     1608.0   448.622997     1608.0   \n",
      "9112    207439      285.0   402.692458      285.0   359.975948      285.0   \n",
      "22026   509620     1783.0   608.874779     1783.0   615.502325     1783.0   \n",
      "\n",
      "        POPsum2002  POPct2003   POPsum2003  POPct2004  ...  POPct2007  \\\n",
      "10516   263.698797      581.0   255.382244      581.0  ...      581.0   \n",
      "19503  1449.107211     1020.0  1428.827351     1020.0  ...     1020.0   \n",
      "19991    89.396302      178.0    91.838653      178.0  ...      178.0   \n",
      "7741    442.898423     1210.0   452.101140     1210.0  ...     1210.0   \n",
      "16823   415.927673      290.0   413.175133      290.0  ...      290.0   \n",
      "19626   236.451380     1099.0   227.536741     1099.0  ...     1099.0   \n",
      "8238    359.732860      639.0   364.489421      639.0  ...      639.0   \n",
      "16857   438.400427     1608.0   425.020034     1608.0  ...     1608.0   \n",
      "9112    362.877608      285.0   358.705770      285.0  ...      285.0   \n",
      "22026   631.507718     1783.0   697.948199     1783.0  ...     1783.0   \n",
      "\n",
      "        POPsum2007  POPct2008   POPsum2008  POPct2009   POPsum2009  POPct2010  \\\n",
      "10516   251.026839      581.0   246.525195      581.0   254.972028      581.0   \n",
      "19503  1465.019486     1020.0  1530.001588     1020.0  1563.835321     1020.0   \n",
      "19991   108.873826      178.0   112.147379      178.0   112.654012      178.0   \n",
      "7741    472.329304     1210.0   460.445041     1210.0   460.119139     1210.0   \n",
      "16823   484.964576      290.0   441.565822      290.0   465.028605      290.0   \n",
      "19626   264.016619     1099.0   236.561671     1099.0   267.766691     1099.0   \n",
      "8238    374.110727      639.0   363.206374      639.0   359.496408      639.0   \n",
      "16857   454.885641     1608.0   474.350865     1608.0   486.005105     1608.0   \n",
      "9112    364.059147      285.0   359.225066      285.0   385.879790      285.0   \n",
      "22026   795.334416     1783.0   856.485372     1783.0   863.998451     1783.0   \n",
      "\n",
      "        POPsum2010  POPct2011   POPsum2011  \n",
      "10516   253.560906      581.0   248.765749  \n",
      "19503  1638.306114     1020.0  1577.900975  \n",
      "19991   111.694849      178.0   110.787827  \n",
      "7741    465.476650     1210.0   462.665436  \n",
      "16823   517.623194      290.0   522.125666  \n",
      "19626   271.518028     1099.0   277.809175  \n",
      "8238    381.413758      639.0   373.571030  \n",
      "16857   485.211541     1608.0   473.473465  \n",
      "9112    381.755890      285.0   421.031017  \n",
      "22026   901.280603     1783.0   915.982062  \n",
      "\n",
      "[10 rows x 25 columns]\n",
      "Loading data for Msk_2012.tif. Sun Feb 19 13:51:35 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2012. Sun Feb 19 13:53:10 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 13:54:51 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 13:54:55 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 14:07:06 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "15257028  0.381379   453647\n",
      "18891686  0.224191   520302\n",
      "23419311  1.315377   401399\n",
      "37647321  0.322855    37188\n",
      "11643287  0.358205   581492\n",
      "8027461   0.440533   581601\n",
      "24047816  0.336992   400480\n",
      "41286101  0.441702   247503\n",
      "18460284  0.191699   417014\n",
      "21455986  0.817667   457284\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Exported as table. Sun Feb 19 14:07:55 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 14:07:55 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2012. Sun Feb 19 14:07:56 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "7584    185283      121.0   40.952139      121.0   39.569841      121.0   \n",
      "23425   597640     1492.0  169.797661     1492.0  138.140194     1492.0   \n",
      "19570   458222      866.0  250.778874      866.0  304.515543      866.0   \n",
      "5621    137780     1718.0  540.643062     1718.0  501.669469     1718.0   \n",
      "3970     93590      272.0  207.007109      272.0  208.928479      272.0   \n",
      "18809   457273      214.0  192.246790      214.0  238.644037      214.0   \n",
      "8933    204569      700.0  488.239304      700.0  553.851144      700.0   \n",
      "8280    203318      420.0  489.551864      420.0  442.361914      420.0   \n",
      "11242   250165      683.0  369.296495      683.0  433.433566      683.0   \n",
      "22444   528736      993.0  125.535631      993.0  145.597178      993.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  ...  POPct2008  \\\n",
      "7584    41.109522      121.0   42.086719      121.0  ...      121.0   \n",
      "23425  135.162521     1492.0  146.481257     1492.0  ...     1492.0   \n",
      "19570  276.059225      866.0  282.698739      866.0  ...      866.0   \n",
      "5621   528.725772     1718.0  540.530213     1718.0  ...     1718.0   \n",
      "3970   203.128499      272.0  193.785951      272.0  ...      272.0   \n",
      "18809  218.647219      214.0  220.270565      214.0  ...      214.0   \n",
      "8933   578.873977      700.0  596.552646      700.0  ...      700.0   \n",
      "8280   434.238296      420.0  449.611056      420.0  ...      420.0   \n",
      "11242  430.073045      683.0  463.941850      683.0  ...      683.0   \n",
      "22444  158.850600      993.0  164.107339      993.0  ...      993.0   \n",
      "\n",
      "       POPsum2008  POPct2009  POPsum2009  POPct2010  POPsum2010  POPct2011  \\\n",
      "7584    66.205966      121.0   62.513465      121.0   65.888621      121.0   \n",
      "23425  164.456050     1492.0  162.759379     1492.0  175.727336     1492.0   \n",
      "19570  318.617322      866.0  344.166702      866.0  368.307477      866.0   \n",
      "5621   615.073463     1718.0  626.389179     1718.0  630.041680     1718.0   \n",
      "3970   209.501188      272.0  217.376912      272.0  208.751949      272.0   \n",
      "18809  218.370799      214.0  234.269812      214.0  240.395566      214.0   \n",
      "8933   552.168355      700.0  601.604819      700.0  600.983172      700.0   \n",
      "8280   442.706173      420.0  464.477919      420.0  489.099538      420.0   \n",
      "11242  648.631108      683.0  645.245328      683.0  683.999690      683.0   \n",
      "22444  189.639655      993.0  201.646010      993.0  210.667435      993.0   \n",
      "\n",
      "       POPsum2011  POPct2012  POPsum2012  \n",
      "7584    69.386340      121.0   80.180273  \n",
      "23425  175.201153     1492.0  169.738605  \n",
      "19570  374.898031      866.0  402.221387  \n",
      "5621   626.784970     1718.0  605.209586  \n",
      "3970   257.638075      272.0  288.808524  \n",
      "18809  242.691946      214.0  251.191480  \n",
      "8933   642.366705      700.0  664.042900  \n",
      "8280   473.993090      420.0  485.816457  \n",
      "11242  709.979680      683.0  596.583008  \n",
      "22444  229.035519      993.0  240.494685  \n",
      "\n",
      "[10 rows x 27 columns]\n",
      "Loading data for Msk_2013.tif. Sun Feb 19 14:07:57 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2013. Sun Feb 19 14:09:41 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 14:11:34 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 14:11:39 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 14:24:04 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "31701614  1.277645   180085\n",
      "26307470  0.413832   388678\n",
      "32403654  1.142859   294532\n",
      "41095761  0.385872   188439\n",
      "13543314  0.229232   475591\n",
      "16802534  0.135129   568887\n",
      "13634859  0.216088   581255\n",
      "33777541  0.306744   339206\n",
      "27029867  0.721598   527973\n",
      "37015482  1.821757   248029\n",
      "\n",
      "Exported as table. Sun Feb 19 14:24:51 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 14:24:52 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2013. Sun Feb 19 14:24:52 2023 \n",
      "\n",
      "       Sett_ID  POPct2000   POPsum2000  POPct2001   POPsum2001  POPct2002  \\\n",
      "5982    147672      305.0   213.582114      305.0   219.506154      305.0   \n",
      "4215     98263      373.0   179.920162      373.0   202.936715      373.0   \n",
      "4640    117191     2479.0  1610.652302     2479.0  1453.075628     2479.0   \n",
      "2511     51663      595.0  1603.433857      595.0  1558.298168      595.0   \n",
      "14841   400708     1450.0  1547.475618     1450.0  1597.430857     1450.0   \n",
      "22180   527740      485.0   337.067550      485.0   411.612435      485.0   \n",
      "5386    129914     1039.0   352.272267     1039.0   324.316315     1039.0   \n",
      "16785   417239     1678.0   411.287248     1678.0   398.597365     1678.0   \n",
      "3784     87519      506.0   112.029906      506.0   107.316983      506.0   \n",
      "8366    203456      426.0   401.845812      426.0   371.489250      426.0   \n",
      "\n",
      "        POPsum2002  POPct2003   POPsum2003  POPct2004  ...  POPct2009  \\\n",
      "5982    214.266588      305.0   216.668126      305.0  ...      305.0   \n",
      "4215    198.286115      373.0   204.769030      373.0  ...      373.0   \n",
      "4640   1475.775001     2479.0  1437.203806     2479.0  ...     2479.0   \n",
      "2511   1577.701633      595.0  1533.150150      595.0  ...      595.0   \n",
      "14841  1534.700340     1450.0  1454.891134     1450.0  ...     1450.0   \n",
      "22180   381.188411      485.0   402.174023      485.0  ...      485.0   \n",
      "5386    321.067505     1039.0   449.856159     1039.0  ...     1039.0   \n",
      "16785   425.775298     1678.0   450.101186     1678.0  ...     1678.0   \n",
      "3784    108.660471      506.0   113.736519      506.0  ...      506.0   \n",
      "8366    377.356715      426.0   381.922281      426.0  ...      426.0   \n",
      "\n",
      "        POPsum2009  POPct2010   POPsum2010  POPct2011   POPsum2011  POPct2012  \\\n",
      "5982    252.644498      305.0   249.584203      305.0   254.596574      305.0   \n",
      "4215    210.117685      373.0   198.961643      373.0   227.453947      373.0   \n",
      "4640   2496.207742     2479.0  2431.995178     2479.0  2495.404865     2479.0   \n",
      "2511   1716.479310      595.0  1770.617438      595.0  2143.058896      595.0   \n",
      "14841  1572.918691     1450.0  1632.253651     1450.0  1640.970082     1450.0   \n",
      "22180   432.479093      485.0   458.211635      485.0   457.175498      485.0   \n",
      "5386    505.963815     1039.0   519.582038     1039.0   530.536748     1039.0   \n",
      "16785   533.957622     1678.0   537.290366     1678.0   556.722998     1678.0   \n",
      "3784    108.378501      506.0   109.379115      506.0   109.545695      506.0   \n",
      "8366    404.210609      426.0   398.627191      426.0   391.598873      426.0   \n",
      "\n",
      "        POPsum2012  POPct2013   POPsum2013  \n",
      "5982    263.110657      305.0   262.102573  \n",
      "4215    272.023939      373.0   281.765381  \n",
      "4640   2641.167823     2479.0  2577.027298  \n",
      "2511   2404.794720      595.0  2411.626923  \n",
      "14841  1548.323474     1450.0  1582.675026  \n",
      "22180   501.746983      485.0   525.377386  \n",
      "5386    536.932355     1039.0   551.374571  \n",
      "16785   563.381352     1678.0   582.336217  \n",
      "3784    106.948662      506.0   106.181578  \n",
      "8366    379.367483      426.0   379.448293  \n",
      "\n",
      "[10 rows x 29 columns]\n",
      "Loading data for Msk_2014.tif. Sun Feb 19 14:24:53 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2014. Sun Feb 19 14:26:34 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 14:28:23 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 14:28:27 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 14:40:20 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "17400779  0.373036   458028\n",
      "22209028  0.451951   400616\n",
      "47716081  1.167657   155969\n",
      "24885285  0.649305   508975\n",
      "32710367  0.392759   319058\n",
      "17364798  0.274485   458027\n",
      "11013539  0.249837   491157\n",
      "41922760  0.902402   265687\n",
      "38531057  0.621362   266170\n",
      "54738843  0.552016    11019\n",
      "\n",
      "Exported as table. Sun Feb 19 14:41:07 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 14:41:07 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2014. Sun Feb 19 14:41:08 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "16097   415568      531.0  304.680316      531.0  269.936851      531.0   \n",
      "12148   286494      649.0  543.570887      649.0  569.501007      649.0   \n",
      "18646   457064      791.0   56.176403      791.0   63.694635      791.0   \n",
      "17945   453888     1069.0  298.139939     1069.0  267.823247     1069.0   \n",
      "7961    190152     1098.0  123.703601     1098.0  115.646539     1098.0   \n",
      "17705   446014      638.0  107.801946      638.0   99.600412      638.0   \n",
      "17571   445786      213.0  150.428548      213.0  131.207047      213.0   \n",
      "3237     73496      729.0  337.410487      729.0  321.289869      729.0   \n",
      "1201     30000     1108.0  203.613947     1108.0  213.460412     1108.0   \n",
      "22659   568589     1791.0  543.189983     1791.0  591.973740     1791.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  ...  POPct2010  \\\n",
      "16097  275.117175      531.0  288.733061      531.0  ...      531.0   \n",
      "12148  583.838351      649.0  590.748161      649.0  ...      649.0   \n",
      "18646   66.565678      791.0   71.343634      791.0  ...      791.0   \n",
      "17945  268.887163     1069.0  289.510004     1069.0  ...     1069.0   \n",
      "7961   112.660691     1098.0  127.922973     1098.0  ...     1098.0   \n",
      "17705  109.432107      638.0  121.595565      638.0  ...      638.0   \n",
      "17571  135.388601      213.0  138.433899      213.0  ...      213.0   \n",
      "3237   352.376650      729.0  355.046229      729.0  ...      729.0   \n",
      "1201   259.307029     1108.0  258.035126     1108.0  ...     1108.0   \n",
      "22659  606.349683     1791.0  651.875717     1791.0  ...     1791.0   \n",
      "\n",
      "       POPsum2010  POPct2011  POPsum2011  POPct2012   POPsum2012  POPct2013  \\\n",
      "16097  295.093186      531.0  286.766323      531.0   339.243513      531.0   \n",
      "12148  624.369848      649.0  636.464873      649.0   645.005027      649.0   \n",
      "18646   77.352369      791.0   80.579599      791.0    77.463396      791.0   \n",
      "17945  352.349438     1069.0  360.248195     1069.0   347.316888     1069.0   \n",
      "7961   198.931630     1098.0  223.800702     1098.0   223.612659     1098.0   \n",
      "17705  138.747989      638.0  171.224863      638.0   173.661726      638.0   \n",
      "17571  144.453114      213.0  199.289745      213.0   214.635493      213.0   \n",
      "3237   388.297741      729.0  373.069468      729.0   387.616705      729.0   \n",
      "1201   293.020144     1108.0  298.923453     1108.0   301.028360     1108.0   \n",
      "22659  847.639991     1791.0  879.640497     1791.0  1154.795887     1791.0   \n",
      "\n",
      "        POPsum2013  POPct2014   POPsum2014  \n",
      "16097   396.408422      531.0   430.208810  \n",
      "12148   689.557902      649.0   688.279651  \n",
      "18646    81.872873      791.0    84.947602  \n",
      "17945   342.130457     1069.0   374.170217  \n",
      "7961    225.258924     1098.0   221.018766  \n",
      "17705   175.334395      638.0   169.556559  \n",
      "17571   194.187183      213.0   203.035329  \n",
      "3237    376.013556      729.0   426.574572  \n",
      "1201    298.327184     1108.0   288.261415  \n",
      "22659  1110.887599     1791.0  1025.588878  \n",
      "\n",
      "[10 rows x 31 columns]\n",
      "Loading data for Msk_2015.tif. Sun Feb 19 14:41:09 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2015. Sun Feb 19 14:42:49 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 14:44:37 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 14:44:41 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 14:56:43 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "26999670  0.461840   396502\n",
      "21043354  0.309879   457323\n",
      "41187942  1.206055   277700\n",
      "55430876  0.187693    22285\n",
      "55251439  0.183287    22285\n",
      "21348900  0.819138   445259\n",
      "26854790  0.522378   388653\n",
      "33810729  0.574076   215329\n",
      "49069154  0.271103      715\n",
      "16988218  0.933347   445543\n",
      "\n",
      "Exported as table. Sun Feb 19 14:57:30 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 14:57:30 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2015. Sun Feb 19 14:57:31 2023 \n",
      "\n",
      "       Sett_ID  POPct2000  POPsum2000  POPct2001  POPsum2001  POPct2002  \\\n",
      "18255   454238      238.0  199.117647      238.0  203.688511      238.0   \n",
      "18897   457403      543.0  192.867069      543.0  171.680119      543.0   \n",
      "8653    203905      378.0  354.883904      378.0  382.363922      378.0   \n",
      "17430   445558     1175.0  447.740365     1175.0  380.307635     1175.0   \n",
      "1866     37180      489.0  225.074390      489.0  265.650033      489.0   \n",
      "3112     67090      248.0   96.663898      248.0   92.176547      248.0   \n",
      "17211   445265      150.0  145.567066      150.0  175.031678      150.0   \n",
      "16365   416180     1005.0  348.746102     1005.0  407.635167     1005.0   \n",
      "21025   475087      588.0  230.788093      588.0  232.740578      588.0   \n",
      "1980     38991      549.0  592.497752      549.0  751.190582      549.0   \n",
      "\n",
      "       POPsum2002  POPct2003  POPsum2003  POPct2004  ...  POPct2011  \\\n",
      "18255  216.785748      238.0  221.965092      238.0  ...      238.0   \n",
      "18897  172.689990      543.0  184.073214      543.0  ...      543.0   \n",
      "8653   382.899613      378.0  391.980553      378.0  ...      378.0   \n",
      "17430  378.280177     1175.0  387.713631     1175.0  ...     1175.0   \n",
      "1866   254.703482      489.0  257.758821      489.0  ...      489.0   \n",
      "3112    90.426448      248.0   95.947446      248.0  ...      248.0   \n",
      "17211  186.675886      150.0  171.397497      150.0  ...      150.0   \n",
      "16365  415.788575     1005.0  423.884838     1005.0  ...     1005.0   \n",
      "21025  225.928379      588.0  205.201516      588.0  ...      588.0   \n",
      "1980   724.933532      549.0  894.227339      549.0  ...      549.0   \n",
      "\n",
      "        POPsum2011  POPct2012   POPsum2012  POPct2013   POPsum2013  POPct2014  \\\n",
      "18255   248.197502      238.0   249.709548      238.0   258.945874      238.0   \n",
      "18897   218.755854      543.0   226.450785      543.0   231.828813      543.0   \n",
      "8653    523.742795      378.0   534.711138      378.0   544.346188      378.0   \n",
      "17430   412.425567     1175.0   387.746759     1175.0   399.596573     1175.0   \n",
      "1866    400.422185      489.0   432.655767      489.0   465.026630      489.0   \n",
      "3112    180.735026      248.0   202.828807      248.0   213.597090      248.0   \n",
      "17211   219.333649      150.0   205.990345      150.0   212.432051      150.0   \n",
      "16365   563.461598     1005.0   587.803369     1005.0   597.991943     1005.0   \n",
      "21025   273.433276      588.0   268.210970      588.0   272.510275      588.0   \n",
      "1980   1132.790243      549.0  1085.649512      549.0  1184.686441      549.0   \n",
      "\n",
      "        POPsum2014  POPct2015   POPsum2015  \n",
      "18255   244.057961      238.0   247.342122  \n",
      "18897   260.282941      543.0   264.278444  \n",
      "8653    594.326986      378.0   584.291125  \n",
      "17430   595.524812     1175.0   607.856631  \n",
      "1866    439.115105      489.0   460.801081  \n",
      "3112    210.219725      248.0   211.188746  \n",
      "17211   204.364541      150.0   207.963038  \n",
      "16365   565.063096     1005.0   580.447972  \n",
      "21025   297.520193      588.0   293.946882  \n",
      "1980   1254.387317      549.0  1276.135312  \n",
      "\n",
      "[10 rows x 33 columns]\n",
      "\n",
      "\n",
      "Finished. All years masked and assigned their nearest settlement. Sun Feb 19 14:57:32 2023\n",
      "       Sett_ID  POPct2000   POPsum2000  POPct2001   POPsum2001  POPct2002  \\\n",
      "7610    185862       56.0    27.097215       56.0    26.987008       56.0   \n",
      "7282    180453      542.0   239.138800      542.0   232.223257      542.0   \n",
      "10725   247095     1648.0   712.643451     1648.0   757.594350     1648.0   \n",
      "7840    188505      573.0   327.903104      573.0   369.124715      573.0   \n",
      "10222   227586      754.0   482.724169      754.0   538.602852      754.0   \n",
      "6198    156009      336.0   166.998720      336.0   155.389841      336.0   \n",
      "20048   464010      251.0   228.721773      251.0   220.093064      251.0   \n",
      "22390   528453     1042.0  1641.962011     1042.0  1489.839201     1042.0   \n",
      "10271   227665      668.0   262.994879      668.0   248.066561      668.0   \n",
      "17522   445674      304.0    73.709315      304.0    70.504793      304.0   \n",
      "\n",
      "        POPsum2002  POPct2003   POPsum2003  POPct2004  ...  POPct2011  \\\n",
      "7610     30.755953       56.0    32.106285       56.0  ...       56.0   \n",
      "7282    232.757708      542.0   230.271442      542.0  ...      542.0   \n",
      "10725   744.600732     1648.0   711.597354     1648.0  ...     1648.0   \n",
      "7840    367.977988      573.0   393.501873      573.0  ...      573.0   \n",
      "10222   698.340593      754.0   558.062472      754.0  ...      754.0   \n",
      "6198    158.150913      336.0   159.348586      336.0  ...      336.0   \n",
      "20048   220.123188      251.0   223.445339      251.0  ...      251.0   \n",
      "22390  1574.437543     1042.0  1676.264952     1042.0  ...     1042.0   \n",
      "10271   256.373846      668.0   246.866327      668.0  ...      668.0   \n",
      "17522    73.227836      304.0    75.181840      304.0  ...      304.0   \n",
      "\n",
      "        POPsum2011  POPct2012   POPsum2012  POPct2013   POPsum2013  POPct2014  \\\n",
      "7610     35.316900       56.0    35.956039       56.0    34.899321       56.0   \n",
      "7282    328.356981      542.0   324.515621      542.0   333.019538      542.0   \n",
      "10725   709.777764     1648.0   695.922541     1648.0   706.232458     1648.0   \n",
      "7840    455.949190      573.0   516.401666      573.0   537.184942      573.0   \n",
      "10222   968.931129      754.0   994.527890      754.0  1079.465640      754.0   \n",
      "6198    148.251682      336.0   145.326803      336.0   138.428771      336.0   \n",
      "20048   236.768786      251.0   273.497335      251.0   245.262633      251.0   \n",
      "22390  1993.153815     1042.0  2254.279492     1042.0  2164.066594     1042.0   \n",
      "10271   237.829586      668.0   220.705361      668.0   200.592685      668.0   \n",
      "17522    80.990549      304.0    78.313625      304.0    79.901761      304.0   \n",
      "\n",
      "        POPsum2014  POPct2015   POPsum2015  \n",
      "7610     32.263909       56.0    33.466426  \n",
      "7282    433.605539      542.0   328.127828  \n",
      "10725   665.390952     1648.0   711.430502  \n",
      "7840    482.035255      573.0   508.467100  \n",
      "10222  1024.528453      754.0  1179.787669  \n",
      "6198    127.904870      336.0   128.282203  \n",
      "20048   234.769698      251.0   243.062700  \n",
      "22390  2034.227430     1042.0  2242.132677  \n",
      "10271   256.843249      668.0   255.926940  \n",
      "17522    84.345134      304.0    85.513035  \n",
      "\n",
      "[10 rows x 33 columns]\n",
      "Saved to file. Sun Feb 19 14:57:32 2023 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "BatchZonalStats(FolderName='Population', Zones=Settlements, StatsWanted=['count', 'sum'], SeriesStart=2000, SeriesEnd=2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47a37640",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e49e4cf",
   "metadata": {},
   "source": [
    "## 9. PREPARE YEARLY DATASETS: NIGHTTIME LIGHTS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4667dd97",
   "metadata": {},
   "source": [
    "### 9.1 Reclassify with settlement buffer mask.\n",
    "Reclassify so that we only need to work with cells within X distance of settlements. The two NTL sources have already been reprojected in a separate script, and cropped to Central & Western Africa."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "5581ef17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['D_1999_avg.tif', 'D_1999_cfc.tif', 'D_2000_avg.tif', 'D_2000_cfc.tif', 'D_2001_avg.tif', 'D_2001_cfc.tif', 'D_2002_avg.tif', 'D_2002_cfc.tif', 'D_2003_avg.tif', 'D_2003_cfc.tif', 'D_2004_avg.tif', 'D_2004_cfc.tif', 'D_2005_avg.tif', 'D_2005_cfc.tif', 'D_2006_avg.tif', 'D_2006_cfc.tif', 'D_2007_avg.tif', 'D_2007_cfc.tif', 'D_2008_avg.tif', 'D_2008_cfc.tif', 'D_2009_avg.tif', 'D_2009_cfc.tif', 'D_2010_avg.tif', 'D_2010_cfc.tif', 'D_2011_avg.tif', 'D_2011_cfc.tif', 'D_2012_avg.tif', 'D_2012_cfc.tif', 'D_2013_avg.tif', 'D_2013_cfc.tif', 'V_2012_avg.tif', 'V_2012_cfc.tif', 'V_2013_avg.tif', 'V_2013_cfc.tif', 'V_2014_avg.tif', 'V_2014_cfc.tif', 'V_2015_avg.tif', 'V_2015_cfc.tif']\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:04 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_1999_avg.tif. Sun Feb 19 10:19:07 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:07 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:07 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:08 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_1999_cfc.tif. Sun Feb 19 10:19:10 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:10 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:10 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:12 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2000_avg.tif. Sun Feb 19 10:19:15 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:15 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:15 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:16 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2000_cfc.tif. Sun Feb 19 10:19:18 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:18 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:18 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:20 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2001_avg.tif. Sun Feb 19 10:19:23 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:23 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:23 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:23 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2001_cfc.tif. Sun Feb 19 10:19:26 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:26 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:26 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:28 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2002_avg.tif. Sun Feb 19 10:19:31 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:31 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:31 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:31 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2002_cfc.tif. Sun Feb 19 10:19:35 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:35 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:35 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:36 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2003_avg.tif. Sun Feb 19 10:19:39 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:39 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:39 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:40 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2003_cfc.tif. Sun Feb 19 10:19:43 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:43 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:43 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:44 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2004_avg.tif. Sun Feb 19 10:19:47 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:47 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:47 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:48 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2004_cfc.tif. Sun Feb 19 10:19:50 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:50 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:50 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:52 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2005_avg.tif. Sun Feb 19 10:19:55 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:55 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:55 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:19:56 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2005_cfc.tif. Sun Feb 19 10:19:59 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:19:59 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:19:59 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:00 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2006_avg.tif. Sun Feb 19 10:20:03 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:03 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:03 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:04 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2006_cfc.tif. Sun Feb 19 10:20:07 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:07 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:07 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:08 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2007_avg.tif. Sun Feb 19 10:20:11 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:11 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:11 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:12 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2007_cfc.tif. Sun Feb 19 10:20:15 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:15 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:15 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:16 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2008_avg.tif. Sun Feb 19 10:20:19 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:19 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:19 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:20 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished rasterio.mask.mask() for D_2008_cfc.tif. Sun Feb 19 10:20:23 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:23 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:23 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:24 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2009_avg.tif. Sun Feb 19 10:20:27 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:27 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:27 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:28 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2009_cfc.tif. Sun Feb 19 10:20:30 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:30 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:30 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:31 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2010_avg.tif. Sun Feb 19 10:20:34 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:34 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:34 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:35 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2010_cfc.tif. Sun Feb 19 10:20:38 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:38 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:38 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:39 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2011_avg.tif. Sun Feb 19 10:20:41 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:41 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:41 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:42 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2011_cfc.tif. Sun Feb 19 10:20:45 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:45 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:45 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:46 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2012_avg.tif. Sun Feb 19 10:20:49 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:49 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:49 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:50 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2012_cfc.tif. Sun Feb 19 10:20:53 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:53 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:53 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:54 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2013_avg.tif. Sun Feb 19 10:20:57 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:20:57 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:20:57 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:20:58 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for D_2013_cfc.tif. Sun Feb 19 10:21:01 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:01 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:01 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:05 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2012_avg.tif. Sun Feb 19 10:21:08 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:08 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:08 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:12 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2012_cfc.tif. Sun Feb 19 10:21:15 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:15 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:15 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:19 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2013_avg.tif. Sun Feb 19 10:21:23 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:23 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:23 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:26 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2013_cfc.tif. Sun Feb 19 10:21:29 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:29 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:29 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:33 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2014_avg.tif. Sun Feb 19 10:21:37 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:37 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:37 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:40 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2014_cfc.tif. Sun Feb 19 10:21:43 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:44 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:44 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:48 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2015_avg.tif. Sun Feb 19 10:21:51 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:51 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:51 2023 \n",
      "\n",
      "Source projection:  None\n",
      "Destination projection:  Africa_Albers_Equal_Area_Conic\n",
      "Finished gdal.Warp() for Msk_D_1999_avg.tif. Sun Feb 19 10:21:54 2023 \n",
      "\n",
      "We warped the data, so we'll use that file for next step.\n",
      "Finished rasterio.mask.mask() for V_2015_cfc.tif. Sun Feb 19 10:21:58 2023 \n",
      "\n",
      "Written to file. Sun Feb 19 10:21:58 2023 \n",
      "\n",
      "Removed intermediate file. Sun Feb 19 10:21:58 2023 \n",
      "\n",
      "\n",
      " \n",
      " Finished all years in list. Sun Feb 19 10:21:58 2023\n"
     ]
    }
   ],
   "source": [
    "MaskByZone(MaskPath = r'Results/Catchment.gpkg', MaskLayerName = 'Buff250m_2015', \n",
    "          SourceFolder = r\"NTL/SourceFiles\", DestFolder = 'NTL')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "8762b626",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Msk_D_1999_avg.tif', 'Msk_D_1999_cfc.tif', 'Msk_D_2000_avg.tif', 'Msk_D_2000_cfc.tif', 'Msk_D_2001_avg.tif', 'Msk_D_2001_cfc.tif', 'Msk_D_2002_avg.tif', 'Msk_D_2002_cfc.tif', 'Msk_D_2003_avg.tif', 'Msk_D_2003_cfc.tif', 'Msk_D_2004_avg.tif', 'Msk_D_2004_cfc.tif', 'Msk_D_2005_avg.tif', 'Msk_D_2005_cfc.tif', 'Msk_D_2006_avg.tif', 'Msk_D_2006_cfc.tif', 'Msk_D_2007_avg.tif', 'Msk_D_2007_cfc.tif', 'Msk_D_2008_avg.tif', 'Msk_D_2008_cfc.tif', 'Msk_D_2009_avg.tif', 'Msk_D_2009_cfc.tif', 'Msk_D_2010_avg.tif', 'Msk_D_2010_cfc.tif', 'Msk_D_2011_avg.tif', 'Msk_D_2011_cfc.tif', 'Msk_D_2012_avg.tif', 'Msk_D_2012_cfc.tif', 'Msk_D_2013_avg.tif', 'Msk_D_2013_cfc.tif', 'Msk_V_2012_avg.tif', 'Msk_V_2012_cfc.tif', 'Msk_V_2013_avg.tif', 'Msk_V_2013_cfc.tif', 'Msk_V_2014_avg.tif', 'Msk_V_2014_cfc.tif', 'Msk_V_2015_avg.tif', 'Msk_V_2015_cfc.tif', 'PreDefiningFunction', 'SourceFiles']\n"
     ]
    }
   ],
   "source": [
    "print(os.listdir('NTL/'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c52640a6",
   "metadata": {},
   "source": [
    "### 9.2 Raster values summarized by settlement.\n",
    "1. Convert each annualized raster to .xyz, \n",
    "2. then bring them to vector space and assign their Sett_ID,\n",
    "3. and finally, aggregate the value as appropriate to the settlement level and save table to file.\n",
    "\n",
    "XYZ is similar to .csv. Raster cell centers are stored as x and y, and their value is stored as z."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "3e920861",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS_equalarea')[['Sett_ID', 'geometry']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "096f5819",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Msk_D_1999_avg.tif', 'Msk_D_1999_cfc.tif', 'Msk_D_2000_avg.tif', 'Msk_D_2000_cfc.tif', 'Msk_D_2001_avg.tif', 'Msk_D_2001_cfc.tif', 'Msk_D_2002_avg.tif', 'Msk_D_2002_cfc.tif', 'Msk_D_2003_avg.tif', 'Msk_D_2003_cfc.tif', 'Msk_D_2004_avg.tif', 'Msk_D_2004_cfc.tif', 'Msk_D_2005_avg.tif', 'Msk_D_2005_cfc.tif', 'Msk_D_2006_avg.tif', 'Msk_D_2006_cfc.tif', 'Msk_D_2007_avg.tif', 'Msk_D_2007_cfc.tif', 'Msk_D_2008_avg.tif', 'Msk_D_2008_cfc.tif', 'Msk_D_2009_avg.tif', 'Msk_D_2009_cfc.tif', 'Msk_D_2010_avg.tif', 'Msk_D_2010_cfc.tif', 'Msk_D_2011_avg.tif', 'Msk_D_2011_cfc.tif', 'Msk_D_2012_avg.tif', 'Msk_D_2012_cfc.tif', 'Msk_D_2013_avg.tif', 'Msk_D_2013_cfc.tif', 'Msk_V_2012_avg.tif', 'Msk_V_2012_cfc.tif', 'Msk_V_2013_avg.tif', 'Msk_V_2013_cfc.tif', 'Msk_V_2014_avg.tif', 'Msk_V_2014_cfc.tif', 'Msk_V_2015_avg.tif', 'Msk_V_2015_cfc.tif']\n",
      "       Sett_ID\n",
      "0            1\n",
      "1            2\n",
      "2            3\n",
      "3            4\n",
      "4            5\n",
      "...        ...\n",
      "23485   612616\n",
      "23486   612723\n",
      "23487   612934\n",
      "23488   612944\n",
      "23489   615066\n",
      "\n",
      "[23490 rows x 1 columns]\n",
      "Loading data for Msk_D_1999_avg.tif. Sun Feb 19 10:25:22 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 1999. Sun Feb 19 10:25:24 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:25:25 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:25:25 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:25:54 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "365558  3.402823e+38   319223\n",
      "380229  3.402823e+38    37192\n",
      "329811  3.402823e+38    37264\n",
      "383107  3.402823e+38   110952\n",
      "170327  3.402823e+38   388831\n",
      "654320  3.402823e+38       53\n",
      "54162   3.402823e+38   388946\n",
      "306097  3.402823e+38   345695\n",
      "220757  3.402823e+38   117674\n",
      "265518  3.402823e+38   527797\n",
      "\n",
      "Exported as table. Sun Feb 19 10:25:56 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:25:56 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 1999. Sun Feb 19 10:25:56 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "4271    100232         19.0  6.125082e+39  3.223727e+38  3.402823e+38   \n",
      "3172     70840          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "13103   310281         13.0  4.083388e+39  3.141068e+38  3.402823e+38   \n",
      "15002   404333          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "19041   457578          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "8845    204346         13.0  3.743106e+39  2.879312e+38  3.402823e+38   \n",
      "6446    160321          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "11140   248050          1.0  2.816264e+00  2.816264e+00  2.816264e+00   \n",
      "22294   528097         73.0  2.484061e+40  3.402823e+38  3.402823e+38   \n",
      "7204    180356         48.0  1.633355e+40  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  \n",
      "4271   2.704167e+00  \n",
      "3172   3.402823e+38  \n",
      "13103  2.759764e+00  \n",
      "15002  2.692029e+00  \n",
      "19041  3.402823e+38  \n",
      "8845   2.926136e+00  \n",
      "6446   2.841667e+00  \n",
      "11140  2.816264e+00  \n",
      "22294  3.402823e+38  \n",
      "7204   3.402823e+38  \n",
      "Loading data for Msk_D_1999_cfc.tif. Sun Feb 19 10:25:56 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 1999. Sun Feb 19 10:25:57 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:25:58 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:25:58 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:26:26 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "635428  255    87183\n",
      "9754    255   597561\n",
      "214238  255   568724\n",
      "166946  255   400834\n",
      "139620  255   576831\n",
      "216916  255   528621\n",
      "635071  255   351448\n",
      "525714  255   165792\n",
      "295330  255   556912\n",
      "466781  255    36896\n",
      "\n",
      "Exported as table. Sun Feb 19 10:26:27 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:26:27 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 1999. Sun Feb 19 10:26:27 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "14675   400494         97.0  3.232682e+40  3.332662e+38  3.402823e+38   \n",
      "20488   472299          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "6092    155861         27.0  8.847341e+39  3.276793e+38  3.402823e+38   \n",
      "12295   294473          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "8650    203899          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "13653   351281         22.0  7.145929e+39  3.248150e+38  3.402823e+38   \n",
      "2402     51505          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "14316   392365         35.0  1.156960e+40  3.305600e+38  3.402823e+38   \n",
      "17850   453763         15.0  3.743106e+39  2.495404e+38  3.402823e+38   \n",
      "3893     92386          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  \n",
      "14675  2.934496e+00     250.835052  \n",
      "20488  3.402823e+38     255.000000  \n",
      "6092   3.176545e+00     247.444444  \n",
      "12295  2.772398e+00     221.833333  \n",
      "8650   3.402823e+38     255.000000  \n",
      "13653  2.524292e+00     245.954545  \n",
      "2402   2.552231e+00     229.250000  \n",
      "14316  2.678184e+00     249.200000  \n",
      "17850  2.460000e+00     202.133333  \n",
      "3893   3.402823e+38     255.000000  \n",
      "Loading data for Msk_D_2000_avg.tif. Sun Feb 19 10:26:27 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2000. Sun Feb 19 10:26:29 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:26:30 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:26:30 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:26:58 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "503871  3.402823e+38    98125\n",
      "345296  3.402823e+38   137684\n",
      "235808  3.402823e+38   547406\n",
      "490476  3.402823e+38   165723\n",
      "60782   3.402823e+38   474538\n",
      "552936  3.402823e+38   249863\n",
      "17044   3.402823e+38   597561\n",
      "618608  3.402823e+38   236577\n",
      "119590  3.402823e+38   581326\n",
      "308858  3.402823e+38   556909\n",
      "\n",
      "Exported as table. Sun Feb 19 10:27:00 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:27:00 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2000. Sun Feb 19 10:27:00 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "13614   345695        101.0  3.436852e+40  3.402823e+38  3.402823e+38   \n",
      "1988     39433         21.0  6.805647e+39  3.240784e+38  3.402823e+38   \n",
      "19093   457639          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "21725   508791         40.0  1.327101e+40  3.317753e+38  3.402823e+38   \n",
      "20864   474781         16.0  5.104235e+39  3.190147e+38  3.402823e+38   \n",
      "15837   409799          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "17491   445635          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "14107   388748         29.0  9.187623e+39  3.168146e+38  3.402823e+38   \n",
      "1980     38991          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "5557    137703         11.0  3.743106e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "13614  3.402823e+38     255.000000        101.0  3.436852e+40  3.402823e+38   \n",
      "1988   2.953733e+00     245.333333         21.0  6.805647e+39  3.240784e+38   \n",
      "19093  2.431173e+00     156.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "21725  2.681818e+00     249.975000         40.0  1.327101e+40  3.317753e+38   \n",
      "20864  2.635873e+00     242.625000         16.0  5.104235e+39  3.190147e+38   \n",
      "15837  2.491587e+00     189.333333          3.0  6.805647e+38  2.268549e+38   \n",
      "17491  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "14107  2.622494e+00     241.000000         29.0  9.187623e+39  3.168146e+38   \n",
      "1980   3.597161e+00     221.333333          6.0  1.701412e+39  2.835686e+38   \n",
      "5557   3.402823e+38     255.000000         11.0  3.743106e+39  3.402823e+38   \n",
      "\n",
      "       NTLmaxD_2000  NTLminD_2000  \n",
      "13614  3.402823e+38  3.402823e+38  \n",
      "1988   3.402823e+38  3.125758e+00  \n",
      "19093  3.402823e+38  2.552778e+00  \n",
      "21725  3.402823e+38  2.577381e+00  \n",
      "20864  3.402823e+38  2.830208e+00  \n",
      "15837  3.402823e+38  2.872424e+00  \n",
      "17491  3.402823e+38  3.402823e+38  \n",
      "14107  3.402823e+38  2.617402e+00  \n",
      "1980   3.402823e+38  3.983871e+00  \n",
      "5557   3.402823e+38  3.402823e+38  \n",
      "Loading data for Msk_D_2000_cfc.tif. Sun Feb 19 10:27:00 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2000. Sun Feb 19 10:27:01 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:27:02 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:27:02 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:27:30 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "298134  255   359197\n",
      "568462  255   351448\n",
      "92607   255   595231\n",
      "239063   64   400517\n",
      "370245  255   366201\n",
      "457103  255   156058\n",
      "84749   255   388932\n",
      "109355   67   468224\n",
      "262767  255   515632\n",
      "526269   62    13849\n",
      "\n",
      "Exported as table. Sun Feb 19 10:27:32 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:27:32 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2000. Sun Feb 19 10:27:32 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "14433   396424          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "5805    138060         30.0  9.527906e+39  3.175969e+38  3.402823e+38   \n",
      "14808   400664         10.0  3.402823e+39  3.402823e+38  3.402823e+38   \n",
      "14124   388776         22.0  7.145929e+39  3.248150e+38  3.402823e+38   \n",
      "14426   396414          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "5191    125407          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "977      23505         14.0  4.763953e+39  3.402823e+38  3.402823e+38   \n",
      "1209     30013          7.0  2.381976e+39  3.402823e+38  3.402823e+38   \n",
      "2976     61086          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "4215     98263          4.0  6.805647e+38  1.701412e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "14433  3.016747e+00     186.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "5805   2.767054e+00     241.300000         30.0  9.527906e+39  3.175969e+38   \n",
      "14808  3.402823e+38     255.000000         10.0  3.402823e+39  3.402823e+38   \n",
      "14124  2.789682e+00     245.909091         22.0  7.145929e+39  3.248150e+38   \n",
      "14426  2.868124e+00     155.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "5191   3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "977    3.402823e+38     255.000000         14.0  4.763953e+39  3.402823e+38   \n",
      "1209   3.402823e+38     255.000000          7.0  2.381976e+39  3.402823e+38   \n",
      "2976   3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "4215   2.671310e+00     152.500000          4.0  6.805647e+38  1.701412e+38   \n",
      "\n",
      "       NTLmaxD_2000  NTLminD_2000  NTLcfc_D_2000  \n",
      "14433  3.402823e+38  3.085399e+00     191.000000  \n",
      "5805   3.402823e+38  2.734119e+00     242.400000  \n",
      "14808  3.402823e+38  3.402823e+38     255.000000  \n",
      "14124  3.402823e+38  2.706676e+00     246.454545  \n",
      "14426  3.402823e+38  2.878575e+00     159.500000  \n",
      "5191   3.402823e+38  3.402823e+38     255.000000  \n",
      "977    3.402823e+38  3.402823e+38     255.000000  \n",
      "1209   3.402823e+38  3.402823e+38     255.000000  \n",
      "2976   3.402823e+38  3.402823e+38     255.000000  \n",
      "4215   3.402823e+38  2.799286e+00     159.000000  \n",
      "Loading data for Msk_D_2001_avg.tif. Sun Feb 19 10:27:32 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2001. Sun Feb 19 10:27:33 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:27:34 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:27:35 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:28:03 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "10301   3.402823e+38   474538\n",
      "1574    3.402823e+38   612944\n",
      "280989  3.402823e+38   379038\n",
      "654792  3.402823e+38   351346\n",
      "653771  3.402823e+38   236547\n",
      "518448  3.402823e+38   165802\n",
      "550021  3.402823e+38   168853\n",
      "254864  3.402823e+38   557001\n",
      "207066  3.402823e+38   568746\n",
      "328691  3.402823e+38   351153\n",
      "\n",
      "Exported as table. Sun Feb 19 10:28:05 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:28:05 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2001. Sun Feb 19 10:28:05 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "17025   430644         23.0  7.826494e+39  3.402823e+38  3.402823e+38   \n",
      "19225   457803          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "21145   480279          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "6470    160980         11.0  3.743106e+39  3.402823e+38  3.402823e+38   \n",
      "1742     37036         50.0  1.701412e+40  3.402823e+38  3.402823e+38   \n",
      "23410   597536         13.0  4.083388e+39  3.141068e+38  3.402823e+38   \n",
      "7310    180486          NaN           NaN           NaN           NaN   \n",
      "16783   417233         22.0  7.486212e+39  3.402823e+38  3.402823e+38   \n",
      "17836   453343         14.0  4.763953e+39  3.402823e+38  3.402823e+38   \n",
      "3890     92329          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "17025  3.402823e+38     255.000000         23.0  7.826494e+39  3.402823e+38   \n",
      "19225  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "21145  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "6470   3.402823e+38     255.000000         11.0  3.743106e+39  3.402823e+38   \n",
      "1742   3.402823e+38     255.000000         50.0  1.701412e+40  3.402823e+38   \n",
      "23410  2.588275e+00     240.153846         13.0  4.083388e+39  3.141068e+38   \n",
      "7310            NaN            NaN          NaN           NaN           NaN   \n",
      "16783  3.402823e+38     255.000000         22.0  7.486212e+39  3.402823e+38   \n",
      "17836  3.402823e+38     255.000000         14.0  4.763953e+39  3.402823e+38   \n",
      "3890   2.683398e+00     232.444444          9.0  2.722259e+39  3.024732e+38   \n",
      "\n",
      "       NTLmaxD_2000  NTLminD_2000  NTLcfc_D_2000  NTLctD_2001  NTLsumD_2001  \\\n",
      "17025  3.402823e+38  3.402823e+38     255.000000         23.0  7.826494e+39   \n",
      "19225  3.402823e+38  3.402823e+38     255.000000          1.0  3.402823e+38   \n",
      "21145  3.402823e+38  3.402823e+38     255.000000          2.0  6.805647e+38   \n",
      "6470   3.402823e+38  3.402823e+38     255.000000         11.0  3.743106e+39   \n",
      "1742   3.402823e+38  3.402823e+38     255.000000         50.0  1.701412e+40   \n",
      "23410  3.402823e+38  2.622222e+00     240.153846         13.0  4.083388e+39   \n",
      "7310            NaN           NaN            NaN          NaN           NaN   \n",
      "16783  3.402823e+38  3.402823e+38     255.000000         22.0  7.486212e+39   \n",
      "17836  3.402823e+38  3.402823e+38     255.000000         14.0  4.763953e+39   \n",
      "3890   3.402823e+38  2.645637e+00     233.111111          9.0  2.722259e+39   \n",
      "\n",
      "       NTLavgD_2001  NTLmaxD_2001  NTLminD_2001  \n",
      "17025  3.402823e+38  3.402823e+38  3.402823e+38  \n",
      "19225  3.402823e+38  3.402823e+38  3.402823e+38  \n",
      "21145  3.402823e+38  3.402823e+38  3.402823e+38  \n",
      "6470   3.402823e+38  3.402823e+38  3.402823e+38  \n",
      "1742   3.402823e+38  3.402823e+38  3.402823e+38  \n",
      "23410  3.141068e+38  3.402823e+38  2.827840e+00  \n",
      "7310            NaN           NaN           NaN  \n",
      "16783  3.402823e+38  3.402823e+38  3.402823e+38  \n",
      "17836  3.402823e+38  3.402823e+38  3.402823e+38  \n",
      "3890   3.024732e+38  3.402823e+38  3.040452e+00  \n",
      "Loading data for Msk_D_2001_cfc.tif. Sun Feb 19 10:28:05 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2001. Sun Feb 19 10:28:06 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:28:07 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:28:07 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:28:36 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "52269   255   388932\n",
      "330388  255   309921\n",
      "202561  255   568769\n",
      "111948  255   400843\n",
      "645698  255   247089\n",
      "498436  255    51259\n",
      "395401  255   357062\n",
      "97417   255   388946\n",
      "181379  255   400807\n",
      "178174  255   568642\n",
      "\n",
      "Exported as table. Sun Feb 19 10:28:37 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:28:37 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2001. Sun Feb 19 10:28:37 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "15782   409723          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "3795     87540         17.0  5.784800e+39  3.402823e+38  3.402823e+38   \n",
      "18369   454362          3.0  3.402823e+38  1.134274e+38  3.402823e+38   \n",
      "14465   396475         12.0  3.743106e+39  3.119255e+38  3.402823e+38   \n",
      "5740    137948          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "14699   400525         15.0  4.763953e+39  3.175969e+38  3.402823e+38   \n",
      "4538    102822         54.0  1.803496e+40  3.339808e+38  3.402823e+38   \n",
      "3435     84296         20.0  6.805647e+39  3.402823e+38  3.402823e+38   \n",
      "4052     95636          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "5770    137996         36.0  1.156960e+40  3.213778e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "15782  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "3795   3.402823e+38     255.000000         17.0  5.784800e+39  3.402823e+38   \n",
      "18369  2.293592e+00     120.333333          3.0  3.402823e+38  1.134274e+38   \n",
      "14465  2.922481e+00     238.083333         12.0  3.743106e+39  3.119255e+38   \n",
      "5740   2.510669e+00     229.375000          8.0  2.381976e+39  2.977471e+38   \n",
      "14699  2.779226e+00     241.466667         15.0  4.763953e+39  3.175969e+38   \n",
      "4538   2.827635e+00     251.240741         54.0  1.803496e+40  3.339808e+38   \n",
      "3435   3.402823e+38     255.000000         20.0  6.805647e+39  3.402823e+38   \n",
      "4052   3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "5770   2.381351e+00     243.555556         36.0  1.156960e+40  3.213778e+38   \n",
      "\n",
      "       NTLmaxD_2000  NTLminD_2000  NTLcfc_D_2000  NTLctD_2001  NTLsumD_2001  \\\n",
      "15782  3.402823e+38  3.402823e+38     255.000000          2.0  6.805647e+38   \n",
      "3795   3.402823e+38  3.402823e+38     255.000000         17.0  5.784800e+39   \n",
      "18369  3.402823e+38  2.479361e+00     129.333333          3.0  3.402823e+38   \n",
      "14465  3.402823e+38  2.747242e+00     239.083333         12.0  3.743106e+39   \n",
      "5740   3.402823e+38  2.750000e+00     231.125000          8.0  2.381976e+39   \n",
      "14699  3.402823e+38  2.701954e+00     242.400000         15.0  4.763953e+39   \n",
      "4538   3.402823e+38  3.197881e+00     251.407407         54.0  1.803496e+40   \n",
      "3435   3.402823e+38  3.402823e+38     255.000000         20.0  6.805647e+39   \n",
      "4052   3.402823e+38  3.402823e+38     255.000000          3.0  1.020847e+39   \n",
      "5770   3.402823e+38  2.821359e+00     244.500000         36.0  1.156960e+40   \n",
      "\n",
      "       NTLavgD_2001  NTLmaxD_2001  NTLminD_2001  NTLcfc_D_2001  \n",
      "15782  3.402823e+38  3.402823e+38  3.402823e+38     255.000000  \n",
      "3795   3.402823e+38  3.402823e+38  3.402823e+38     255.000000  \n",
      "18369  1.134274e+38  3.402823e+38  2.647700e+00     124.666667  \n",
      "14465  3.119255e+38  3.402823e+38  3.000000e+00     238.583333  \n",
      "5740   2.977471e+38  3.402823e+38  3.069122e+00     230.500000  \n",
      "14699  3.175969e+38  3.402823e+38  2.879386e+00     242.200000  \n",
      "4538   3.339808e+38  3.402823e+38  2.793035e+00     251.518519  \n",
      "3435   3.402823e+38  3.402823e+38  3.402823e+38     255.000000  \n",
      "4052   3.402823e+38  3.402823e+38  3.402823e+38     255.000000  \n",
      "5770   3.213778e+38  3.402823e+38  2.851852e+00     244.083333  \n",
      "Loading data for Msk_D_2002_avg.tif. Sun Feb 19 10:28:38 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2002. Sun Feb 19 10:28:39 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:28:40 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:28:40 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:29:09 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "612482  3.402823e+38   351346\n",
      "102809  3.402823e+38   388946\n",
      "592090  3.402823e+38     6273\n",
      "26188   3.402823e+38   388946\n",
      "643361  3.402823e+38     1701\n",
      "38010   3.402823e+38   388946\n",
      "409367  3.402823e+38   132479\n",
      "505253  3.402823e+38   351346\n",
      "6055    3.402823e+38   612944\n",
      "458767  3.402823e+38    51179\n",
      "\n",
      "Exported as table. Sun Feb 19 10:29:11 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:29:11 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2002. Sun Feb 19 10:29:11 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "19958   460349          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "11480   265958         10.0  3.062541e+39  3.062541e+38  3.402823e+38   \n",
      "299        348          NaN           NaN           NaN           NaN   \n",
      "1569     36824         29.0  8.507059e+39  2.933469e+38  3.402823e+38   \n",
      "8720    204030         13.0  4.423671e+39  3.402823e+38  3.402823e+38   \n",
      "7551    184262          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "3154     69842          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "11288   258329         24.0  7.826494e+39  3.261039e+38  3.402823e+38   \n",
      "1364     30211          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "14052   388691         22.0  7.145929e+39  3.248150e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "19958  2.502268e+00     189.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "11480  2.611146e+00     235.200000         10.0  3.062541e+39  3.062541e+38   \n",
      "299             NaN            NaN          NaN           NaN           NaN   \n",
      "1569   2.645804e+00     226.793103         29.0  8.507059e+39  2.933469e+38   \n",
      "8720   3.402823e+38     255.000000         13.0  4.423671e+39  3.402823e+38   \n",
      "7551   3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "3154   2.995703e+00     229.750000          8.0  2.381976e+39  2.977471e+38   \n",
      "11288  2.652035e+00     246.916667         24.0  7.826494e+39  3.261039e+38   \n",
      "1364   3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "14052  2.755285e+00     245.818182         22.0  7.145929e+39  3.248150e+38   \n",
      "\n",
      "       ...  NTLsumD_2001  NTLavgD_2001  NTLmaxD_2001  NTLminD_2001  \\\n",
      "19958  ...  6.805647e+38  2.268549e+38  3.402823e+38  2.790057e+00   \n",
      "11480  ...  3.062541e+39  3.062541e+38  3.402823e+38  2.868056e+00   \n",
      "299    ...           NaN           NaN           NaN           NaN   \n",
      "1569   ...  8.507059e+39  2.933469e+38  3.402823e+38  2.920451e+00   \n",
      "8720   ...  4.423671e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "7551   ...  2.722259e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "3154   ...  2.381976e+39  2.977471e+38  3.402823e+38  2.871482e+00   \n",
      "11288  ...  7.826494e+39  3.261039e+38  3.402823e+38  2.993472e+00   \n",
      "1364   ...  1.701412e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "14052  ...  7.145929e+39  3.248150e+38  3.402823e+38  2.958303e+00   \n",
      "\n",
      "       NTLcfc_D_2001  NTLctD_2002  NTLsumD_2002  NTLavgD_2002  NTLmaxD_2002  \\\n",
      "19958     190.000000          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "11480     235.600000         10.0  3.062541e+39  3.062541e+38  3.402823e+38   \n",
      "299              NaN          NaN           NaN           NaN           NaN   \n",
      "1569      227.965517         29.0  8.507059e+39  2.933469e+38  3.402823e+38   \n",
      "8720      255.000000         13.0  4.423671e+39  3.402823e+38  3.402823e+38   \n",
      "7551      255.000000          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "3154      230.625000          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "11288     247.000000         24.0  7.826494e+39  3.261039e+38  3.402823e+38   \n",
      "1364      255.000000          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "14052     246.363636         22.0  7.145929e+39  3.248150e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_2002  \n",
      "19958  2.500000e+00  \n",
      "11480  2.986495e+00  \n",
      "299             NaN  \n",
      "1569   2.942570e+00  \n",
      "8720   3.402823e+38  \n",
      "7551   3.402823e+38  \n",
      "3154   3.286296e+00  \n",
      "11288  2.701428e+00  \n",
      "1364   3.402823e+38  \n",
      "14052  2.539852e+00  \n",
      "\n",
      "[10 rows x 24 columns]\n",
      "Loading data for Msk_D_2002_cfc.tif. Sun Feb 19 10:29:12 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2002. Sun Feb 19 10:29:12 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:29:13 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:29:13 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:29:42 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "233173  255   571922\n",
      "23704   255   468241\n",
      "153832  255   581189\n",
      "523929  255   165792\n",
      "177016  255   457987\n",
      "544362  255    20088\n",
      "18924   255   388932\n",
      "259278  255   528131\n",
      "107149  255   576831\n",
      "53097   255   568902\n",
      "\n",
      "Exported as table. Sun Feb 19 10:29:43 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:29:43 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2002. Sun Feb 19 10:29:43 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "7480    180709          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "22544   534834         17.0  5.444518e+39  3.202657e+38  3.402823e+38   \n",
      "377        439         10.0  3.402823e+39  3.402823e+38  3.402823e+38   \n",
      "22419   528577         86.0  2.926428e+40  3.402823e+38  3.402823e+38   \n",
      "16135   415643          7.0  2.381976e+39  3.402823e+38  3.402823e+38   \n",
      "18404   454516         24.0  8.166776e+39  3.402823e+38  3.402823e+38   \n",
      "12925   309988          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22916   580982         24.0  7.826494e+39  3.261039e+38  3.402823e+38   \n",
      "8226    203227          NaN           NaN           NaN           NaN   \n",
      "12017   277793          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "7480   2.716667e+00     155.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "22544  2.391393e+00     243.235294         17.0  5.444518e+39  3.202657e+38   \n",
      "377    3.402823e+38     255.000000         10.0  3.402823e+39  3.402823e+38   \n",
      "22419  3.402823e+38     255.000000         86.0  2.926428e+40  3.402823e+38   \n",
      "16135  3.402823e+38     255.000000          7.0  2.381976e+39  3.402823e+38   \n",
      "18404  3.402823e+38     255.000000         24.0  8.166776e+39  3.402823e+38   \n",
      "12925  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "22916  2.690000e+00     246.791667         24.0  7.826494e+39  3.261039e+38   \n",
      "8226            NaN            NaN          NaN           NaN           NaN   \n",
      "12017  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2001  NTLmaxD_2001  NTLminD_2001  NTLcfc_D_2001  \\\n",
      "7480   ...  1.701412e+38  3.402823e+38  3.112469e+00     156.500000   \n",
      "22544  ...  3.202657e+38  3.402823e+38  2.536051e+00     243.588235   \n",
      "377    ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "22419  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "16135  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "18404  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "12925  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "22916  ...  3.261039e+38  3.402823e+38  2.693609e+00     247.041667   \n",
      "8226   ...           NaN           NaN           NaN            NaN   \n",
      "12017  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "\n",
      "       NTLctD_2002  NTLsumD_2002  NTLavgD_2002  NTLmaxD_2002  NTLminD_2002  \\\n",
      "7480           2.0  3.402823e+38  1.701412e+38  3.402823e+38  2.729779e+00   \n",
      "22544         17.0  5.444518e+39  3.202657e+38  3.402823e+38  2.398632e+00   \n",
      "377           10.0  3.402823e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22419         86.0  2.926428e+40  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "16135          7.0  2.381976e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "18404         24.0  8.166776e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "12925          1.0  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22916         24.0  7.826494e+39  3.261039e+38  3.402823e+38  2.584445e+00   \n",
      "8226           NaN           NaN           NaN           NaN           NaN   \n",
      "12017          4.0  1.361129e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLcfc_D_2002  \n",
      "7480      157.000000  \n",
      "22544     243.411765  \n",
      "377       255.000000  \n",
      "22419     255.000000  \n",
      "16135     255.000000  \n",
      "18404     255.000000  \n",
      "12925     255.000000  \n",
      "22916     246.875000  \n",
      "8226             NaN  \n",
      "12017     255.000000  \n",
      "\n",
      "[10 rows x 25 columns]\n",
      "Loading data for Msk_D_2003_avg.tif. Sun Feb 19 10:29:43 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2003. Sun Feb 19 10:29:45 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:29:46 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:29:46 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:30:14 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "25463   3.402823e+38   458340\n",
      "225681  3.402823e+38   409860\n",
      "299400  3.402823e+38   138148\n",
      "74127   3.402823e+38   458333\n",
      "253450  3.402823e+38   396608\n",
      "389230  3.402823e+38   358868\n",
      "236563  3.402823e+38   416042\n",
      "593025  3.402823e+38       86\n",
      "474444  3.402823e+38   236577\n",
      "281697  3.402823e+38   508872\n",
      "\n",
      "Exported as table. Sun Feb 19 10:30:16 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:30:16 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2003. Sun Feb 19 10:30:16 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "7047    180155         28.0  9.187623e+39  3.281294e+38  3.402823e+38   \n",
      "10612   236718         22.0  7.486212e+39  3.402823e+38  3.402823e+38   \n",
      "6812    166062         65.0  2.211835e+40  3.402823e+38  3.402823e+38   \n",
      "7726    187408          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "22248   527978          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "22105   519864          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "17765   447200          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "23111   581383          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "15335   409066          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "10630   237846        176.0  5.988969e+40  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "7047   2.775570e+00     247.785714         28.0  9.187623e+39  3.281294e+38   \n",
      "10612  3.402823e+38     255.000000         22.0  7.486212e+39  3.402823e+38   \n",
      "6812   3.402823e+38     255.000000         65.0  2.211835e+40  3.402823e+38   \n",
      "7726   2.810714e+00     221.666667          6.0  1.701412e+39  2.835686e+38   \n",
      "22248  3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "22105  2.489365e+00     233.000000          9.0  2.722259e+39  3.024732e+38   \n",
      "17765  2.622549e+00     158.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "23111  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "15335  2.625432e+00     233.111111          9.0  2.722259e+39  3.024732e+38   \n",
      "10630  3.402823e+38     255.000000        176.0  5.988969e+40  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumD_2002  NTLavgD_2002  NTLmaxD_2002  NTLminD_2002  \\\n",
      "7047   ...  9.187623e+39  3.281294e+38  3.402823e+38  2.616049e+00   \n",
      "10612  ...  7.486212e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "6812   ...  2.211835e+40  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "7726   ...  1.701412e+39  2.835686e+38  3.402823e+38  2.818954e+00   \n",
      "22248  ...  2.722259e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22105  ...  2.722259e+39  3.024732e+38  3.402823e+38  2.563333e+00   \n",
      "17765  ...  3.402823e+38  1.701412e+38  3.402823e+38  2.521739e+00   \n",
      "23111  ...  6.805647e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "15335  ...  2.722259e+39  3.024732e+38  3.402823e+38  2.721257e+00   \n",
      "10630  ...  5.988969e+40  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLcfc_D_2002  NTLctD_2003  NTLsumD_2003  NTLavgD_2003  NTLmaxD_2003  \\\n",
      "7047      247.964286         28.0  9.187623e+39  3.281294e+38  3.402823e+38   \n",
      "10612     255.000000         22.0  7.486212e+39  3.402823e+38  3.402823e+38   \n",
      "6812      255.000000         65.0  2.211835e+40  3.402823e+38  3.402823e+38   \n",
      "7726      222.166667          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "22248     255.000000          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "22105     233.666667          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "17765     157.000000          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "23111     255.000000          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "15335     232.888889          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "10630     255.000000        176.0  5.988969e+40  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_2003  \n",
      "7047   2.036782e+00  \n",
      "10612  3.402823e+38  \n",
      "6812   3.402823e+38  \n",
      "7726   2.166667e+00  \n",
      "22248  3.402823e+38  \n",
      "22105  2.193111e+00  \n",
      "17765  1.934426e+00  \n",
      "23111  3.402823e+38  \n",
      "15335  2.190476e+00  \n",
      "10630  3.402823e+38  \n",
      "\n",
      "[10 rows x 30 columns]\n",
      "Loading data for Msk_D_2003_cfc.tif. Sun Feb 19 10:30:17 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2003. Sun Feb 19 10:30:18 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:30:19 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:30:19 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:30:47 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "134427  255   388946\n",
      "524250  255   351448\n",
      "466997  255   155826\n",
      "633532  255       41\n",
      "399862  255   331856\n",
      "561523  255      555\n",
      "196652  255   400708\n",
      "65436   255   597321\n",
      "224165  255   568681\n",
      "384441  255   247860\n",
      "\n",
      "Exported as table. Sun Feb 19 10:30:48 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:30:48 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2003. Sun Feb 19 10:30:48 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "824      15041          NaN           NaN           NaN           NaN   \n",
      "9762    219490          9.0  3.062541e+39  3.402823e+38  3.402823e+38   \n",
      "3519     85513         17.0  5.444518e+39  3.202657e+38  3.402823e+38   \n",
      "6634    165815         13.0  4.083388e+39  3.141068e+38  3.402823e+38   \n",
      "6679    165868         17.0  5.444518e+39  3.202657e+38  3.402823e+38   \n",
      "216        255          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "18200   454176          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "13031   310148          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "13058   310188          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "4801    117382          1.0  2.964379e+00  2.964379e+00  2.964379e+00   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "824             NaN            NaN          NaN           NaN           NaN   \n",
      "9762   3.402823e+38     255.000000          9.0  3.062541e+39  3.402823e+38   \n",
      "3519   2.713115e+00     242.941176         17.0  5.444518e+39  3.202657e+38   \n",
      "6634   2.621212e+00     239.769231         13.0  4.083388e+39  3.141068e+38   \n",
      "6679   2.702946e+00     243.352941         17.0  5.444518e+39  3.202657e+38   \n",
      "216    3.402823e+38     255.000000          6.0  2.041694e+39  3.402823e+38   \n",
      "18200  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "13031  2.644444e+00     232.555556          9.0  2.722259e+39  3.024732e+38   \n",
      "13058  3.402823e+38     255.000000          6.0  2.041694e+39  3.402823e+38   \n",
      "4801   2.964379e+00      58.000000          1.0  2.756061e+00  2.756061e+00   \n",
      "\n",
      "       ...  NTLavgD_2002  NTLmaxD_2002  NTLminD_2002  NTLcfc_D_2002  \\\n",
      "824    ...           NaN           NaN           NaN            NaN   \n",
      "9762   ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "3519   ...  3.202657e+38  3.402823e+38  2.767673e+00     243.058824   \n",
      "6634   ...  3.141068e+38  3.402823e+38  2.846557e+00     239.769231   \n",
      "6679   ...  3.202657e+38  3.402823e+38  2.952394e+00     243.588235   \n",
      "216    ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "18200  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "13031  ...  3.024732e+38  3.402823e+38  2.546107e+00     232.777778   \n",
      "13058  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "4801   ...  2.806974e+00  2.806974e+00  2.806974e+00      61.000000   \n",
      "\n",
      "       NTLctD_2003  NTLsumD_2003  NTLavgD_2003  NTLmaxD_2003  NTLminD_2003  \\\n",
      "824            NaN           NaN           NaN           NaN           NaN   \n",
      "9762           9.0  3.062541e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "3519          17.0  5.444518e+39  3.202657e+38  3.402823e+38  2.025131e+00   \n",
      "6634          13.0  4.083388e+39  3.141068e+38  3.402823e+38  1.970844e+00   \n",
      "6679          17.0  5.444518e+39  3.202657e+38  3.402823e+38  2.148317e+00   \n",
      "216            6.0  2.041694e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "18200          3.0  1.020847e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "13031          9.0  2.722259e+39  3.024732e+38  3.402823e+38  2.194890e+00   \n",
      "13058          6.0  2.041694e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4801           1.0  2.517045e+00  2.517045e+00  2.517045e+00  2.517045e+00   \n",
      "\n",
      "       NTLcfc_D_2003  \n",
      "824              NaN  \n",
      "9762      255.000000  \n",
      "3519      242.705882  \n",
      "6634      239.000000  \n",
      "6679      242.882353  \n",
      "216       255.000000  \n",
      "18200     255.000000  \n",
      "13031     232.333333  \n",
      "13058     255.000000  \n",
      "4801       52.000000  \n",
      "\n",
      "[10 rows x 31 columns]\n",
      "Loading data for Msk_D_2004_avg.tif. Sun Feb 19 10:30:49 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2004. Sun Feb 19 10:30:50 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:30:51 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:30:51 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:31:20 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "191203  3.402823e+38   388927\n",
      "11851   3.402823e+38   388946\n",
      "57435   3.402823e+38   581615\n",
      "55622   3.402823e+38   597544\n",
      "376151  3.402823e+38   199973\n",
      "41293   3.402823e+38   597561\n",
      "508371  2.796747e+00      773\n",
      "653066  3.402823e+38   351211\n",
      "217010  3.402823e+38   568745\n",
      "77724   3.402823e+38   458333\n",
      "\n",
      "Exported as table. Sun Feb 19 10:31:22 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:31:22 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2004. Sun Feb 19 10:31:22 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "8421    203559          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4467    102720          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "15155   404728          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "21585   502117         11.0  3.743106e+39  3.402823e+38  3.402823e+38   \n",
      "7514    183195          NaN           NaN           NaN           NaN   \n",
      "12509   295037          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4676    117233         10.0  2.722259e+39  2.722259e+38  3.402823e+38   \n",
      "18795   457257          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "16790   417266         16.0  5.444518e+39  3.402823e+38  3.402823e+38   \n",
      "6891    169204          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "8421   3.402823e+38        255.000          1.0  3.402823e+38  3.402823e+38   \n",
      "4467   2.925439e+00        229.625          8.0  2.381976e+39  2.977471e+38   \n",
      "15155  3.402823e+38        255.000          5.0  1.701412e+39  3.402823e+38   \n",
      "21585  3.402823e+38        255.000         11.0  3.743106e+39  3.402823e+38   \n",
      "7514            NaN            NaN          NaN           NaN           NaN   \n",
      "12509  3.402823e+38        255.000          1.0  3.402823e+38  3.402823e+38   \n",
      "4676   2.598438e+00        214.800         10.0  2.722259e+39  2.722259e+38   \n",
      "18795  3.402823e+38        255.000          5.0  1.701412e+39  3.402823e+38   \n",
      "16790  3.402823e+38        255.000         16.0  5.444518e+39  3.402823e+38   \n",
      "6891   3.402823e+38        255.000          6.0  2.041694e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumD_2003  NTLavgD_2003  NTLmaxD_2003  NTLminD_2003  \\\n",
      "8421   ...  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4467   ...  2.381976e+39  2.977471e+38  3.402823e+38  2.232936e+00   \n",
      "15155  ...  1.701412e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "21585  ...  3.743106e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "7514   ...           NaN           NaN           NaN           NaN   \n",
      "12509  ...  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4676   ...  2.722259e+39  2.722259e+38  3.402823e+38  2.334203e+00   \n",
      "18795  ...  1.701412e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "16790  ...  5.444518e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "6891   ...  2.041694e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLcfc_D_2003  NTLctD_2004  NTLsumD_2004  NTLavgD_2004  NTLmaxD_2004  \\\n",
      "8421         255.000          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4467         229.625          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "15155        255.000          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "21585        255.000         11.0  3.743106e+39  3.402823e+38  3.402823e+38   \n",
      "7514             NaN          NaN           NaN           NaN           NaN   \n",
      "12509        255.000          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4676         214.200         10.0  2.722259e+39  2.722259e+38  3.402823e+38   \n",
      "18795        255.000          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "16790        255.000         16.0  5.444518e+39  3.402823e+38  3.402823e+38   \n",
      "6891         255.000          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_2004  \n",
      "8421   3.402823e+38  \n",
      "4467   2.781710e+00  \n",
      "15155  3.402823e+38  \n",
      "21585  3.402823e+38  \n",
      "7514            NaN  \n",
      "12509  3.402823e+38  \n",
      "4676   2.686267e+00  \n",
      "18795  3.402823e+38  \n",
      "16790  3.402823e+38  \n",
      "6891   3.402823e+38  \n",
      "\n",
      "[10 rows x 36 columns]\n",
      "Loading data for Msk_D_2004_cfc.tif. Sun Feb 19 10:31:23 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2004. Sun Feb 19 10:31:24 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:31:25 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:31:25 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:31:54 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "604957  255    84301\n",
      "526491  255    87480\n",
      "395385  255   351319\n",
      "560704  255    87315\n",
      "32039   255   475265\n",
      "465043  255    51458\n",
      "174422  255   417248\n",
      "432387  255   351312\n",
      "592646  255   351346\n",
      "8225    255   388946\n",
      "\n",
      "Exported as table. Sun Feb 19 10:31:55 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:31:55 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2004. Sun Feb 19 10:31:55 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "19319   457918          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "17471   445608          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "15165   404756         14.0  4.763953e+39  3.402823e+38  3.402823e+38   \n",
      "12981   310082          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "2308     51374          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "11556   266126         20.0  6.465365e+39  3.232682e+38  3.402823e+38   \n",
      "15526   409332          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "11723   273306         46.0  1.531271e+40  3.328849e+38  3.402823e+38   \n",
      "5715    137921         14.0  4.423671e+39  3.159765e+38  3.402823e+38   \n",
      "16286   416043          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "19319  2.768939e+00     188.333333          3.0  6.805647e+38  2.268549e+38   \n",
      "17471  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "15165  3.402823e+38     255.000000         14.0  4.763953e+39  3.402823e+38   \n",
      "12981  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "2308   3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "11556  2.888738e+00     245.150000         20.0  6.465365e+39  3.232682e+38   \n",
      "15526  2.571023e+00     156.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "11723  2.777534e+00     250.804348         46.0  1.531271e+40  3.328849e+38   \n",
      "5715   2.804481e+00     240.285714         14.0  4.423671e+39  3.159765e+38   \n",
      "16286  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2003  NTLmaxD_2003  NTLminD_2003  NTLcfc_D_2003  \\\n",
      "19319  ...  2.268549e+38  3.402823e+38  2.179089e+00     187.333333   \n",
      "17471  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "15165  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "12981  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "2308   ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "11556  ...  3.232682e+38  3.402823e+38  2.269913e+00     244.850000   \n",
      "15526  ...  1.701412e+38  3.402823e+38  2.064236e+00     152.500000   \n",
      "11723  ...  3.328849e+38  3.402823e+38  1.815179e+00     250.543478   \n",
      "5715   ...  3.159765e+38  3.402823e+38  2.354342e+00     240.357143   \n",
      "16286  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "\n",
      "       NTLctD_2004  NTLsumD_2004  NTLavgD_2004  NTLmaxD_2004  NTLminD_2004  \\\n",
      "19319          3.0  6.805647e+38  2.268549e+38  3.402823e+38  2.261947e+00   \n",
      "17471          1.0  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "15165         14.0  4.763953e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "12981          1.0  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "2308           5.0  1.701412e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "11556         20.0  6.465365e+39  3.232682e+38  3.402823e+38  2.544796e+00   \n",
      "15526          2.0  3.402823e+38  1.701412e+38  3.402823e+38  2.336663e+00   \n",
      "11723         46.0  1.531271e+40  3.328849e+38  3.402823e+38  2.284616e+00   \n",
      "5715          14.0  4.423671e+39  3.159765e+38  3.402823e+38  2.403718e+00   \n",
      "16286          2.0  6.805647e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLcfc_D_2004  \n",
      "19319     194.000000  \n",
      "17471     255.000000  \n",
      "15165     255.000000  \n",
      "12981     255.000000  \n",
      "2308      255.000000  \n",
      "11556     245.650000  \n",
      "15526     160.500000  \n",
      "11723     250.891304  \n",
      "5715      241.714286  \n",
      "16286     255.000000  \n",
      "\n",
      "[10 rows x 37 columns]\n",
      "Loading data for Msk_D_2005_avg.tif. Sun Feb 19 10:31:56 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2005. Sun Feb 19 10:31:58 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:31:59 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:31:59 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:32:29 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "174647  3.402823e+38   568874\n",
      "506544  3.402823e+38    54050\n",
      "367861  3.402823e+38   129963\n",
      "45882   3.402823e+38   568902\n",
      "213001  3.113179e+00   453764\n",
      "149064  3.402823e+38   468355\n",
      "543533  3.402823e+38    89563\n",
      "434303  3.402823e+38    37031\n",
      "463859  3.402823e+38   351448\n",
      "479889  3.402823e+38   237822\n",
      "\n",
      "Exported as table. Sun Feb 19 10:32:31 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:32:31 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2005. Sun Feb 19 10:32:31 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "11969   277690          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "12208   294277          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "20186   468230          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "20117   467734          1.0  2.285434e+00  2.285434e+00  2.285434e+00   \n",
      "19594   458256         11.0  3.402823e+39  3.093476e+38  3.402823e+38   \n",
      "7682    186771          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "9058    206802          NaN           NaN           NaN           NaN   \n",
      "12280   294446         13.0  4.423671e+39  3.402823e+38  3.402823e+38   \n",
      "17919   453861          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "17777   448074          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "11969  2.765583e+00     155.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "12208  2.716734e+00     188.333333          3.0  6.805647e+38  2.268549e+38   \n",
      "20186  2.833688e+00     154.500000          2.0  3.402823e+38  1.701412e+38   \n",
      "20117  2.285434e+00      56.000000          1.0  2.903968e+00  2.903968e+00   \n",
      "19594  2.460829e+00     236.636364         11.0  3.402823e+39  3.093476e+38   \n",
      "7682   2.662191e+00     155.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "9058            NaN            NaN          NaN           NaN           NaN   \n",
      "12280  3.402823e+38     255.000000         13.0  4.423671e+39  3.402823e+38   \n",
      "17919  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "17777  2.417669e+00     233.222222          9.0  2.722259e+39  3.024732e+38   \n",
      "\n",
      "       ...  NTLsumD_2004  NTLavgD_2004  NTLmaxD_2004  NTLminD_2004  \\\n",
      "11969  ...  3.402823e+38  1.701412e+38  3.402823e+38  2.278555e+00   \n",
      "12208  ...  6.805647e+38  2.268549e+38  3.402823e+38  2.347854e+00   \n",
      "20186  ...  3.402823e+38  1.701412e+38  3.402823e+38  2.336775e+00   \n",
      "20117  ...  2.171459e+00  2.171459e+00  2.171459e+00  2.171459e+00   \n",
      "19594  ...  3.402823e+39  3.093476e+38  3.402823e+38  2.239726e+00   \n",
      "7682   ...  3.402823e+38  1.701412e+38  3.402823e+38  2.559186e+00   \n",
      "9058   ...           NaN           NaN           NaN           NaN   \n",
      "12280  ...  4.423671e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "17919  ...  6.805647e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "17777  ...  2.722259e+39  3.024732e+38  3.402823e+38  2.155329e+00   \n",
      "\n",
      "       NTLcfc_D_2004  NTLctD_2005  NTLsumD_2005  NTLavgD_2005  NTLmaxD_2005  \\\n",
      "11969     161.500000          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "12208     193.000000          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "20186     165.000000          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "20117      72.000000          1.0  1.930171e+00  1.930171e+00  1.930171e+00   \n",
      "19594     238.454545         11.0  3.402823e+39  3.093476e+38  3.402823e+38   \n",
      "7682      160.000000          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "9058             NaN          NaN           NaN           NaN           NaN   \n",
      "12280     255.000000         13.0  4.423671e+39  3.402823e+38  3.402823e+38   \n",
      "17919     255.000000          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "17777     234.000000          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_2005  \n",
      "11969  2.250000e+00  \n",
      "12208  2.036725e+00  \n",
      "20186  1.891952e+00  \n",
      "20117  1.930171e+00  \n",
      "19594  1.927128e+00  \n",
      "7682   2.207126e+00  \n",
      "9058            NaN  \n",
      "12280  3.402823e+38  \n",
      "17919  3.402823e+38  \n",
      "17777  2.042114e+00  \n",
      "\n",
      "[10 rows x 42 columns]\n",
      "Loading data for Msk_D_2005_cfc.tif. Sun Feb 19 10:32:32 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2005. Sun Feb 19 10:32:33 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:32:34 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:32:34 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:33:03 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "563582  255   165795\n",
      "454108  255    40647\n",
      "293325  255   501571\n",
      "614662  255       53\n",
      "470011  255   247304\n",
      "310845  255    37253\n",
      "54260   255   400843\n",
      "461462  255    65053\n",
      "605171  255   249863\n",
      "442062   68   247181\n",
      "\n",
      "Exported as table. Sun Feb 19 10:33:04 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:33:04 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2005. Sun Feb 19 10:33:04 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "11475   265945          7.0  1.701412e+39  2.430588e+38  3.402823e+38   \n",
      "18008   453961          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "16051   415482         40.0  1.327101e+40  3.317753e+38  3.402823e+38   \n",
      "13384   318828         19.0  6.125082e+39  3.223727e+38  3.402823e+38   \n",
      "22332   528179         19.0  6.465365e+39  3.402823e+38  3.402823e+38   \n",
      "10555   236609         14.0  4.423671e+39  3.159765e+38  3.402823e+38   \n",
      "1793     37093          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "59          68         20.0  6.805647e+39  3.402823e+38  3.402823e+38   \n",
      "15653   409549         11.0  3.402823e+39  3.093476e+38  3.402823e+38   \n",
      "16678   416985         12.0  4.083388e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "11475  2.928030e+00     198.428571          7.0  1.701412e+39  2.430588e+38   \n",
      "18008  2.579648e+00     205.750000          4.0  1.020847e+39  2.552118e+38   \n",
      "16051  2.601326e+00     250.050000         40.0  1.327101e+40  3.317753e+38   \n",
      "13384  2.728431e+00     244.526316         19.0  6.125082e+39  3.223727e+38   \n",
      "22332  3.402823e+38     255.000000         19.0  6.465365e+39  3.402823e+38   \n",
      "10555  2.938998e+00     240.857143         14.0  4.423671e+39  3.159765e+38   \n",
      "1793   2.874603e+00     226.142857          7.0  2.041694e+39  2.916706e+38   \n",
      "59     3.402823e+38     255.000000         20.0  6.805647e+39  3.402823e+38   \n",
      "15653  2.369139e+00     237.272727         11.0  3.402823e+39  3.093476e+38   \n",
      "16678  3.402823e+38     255.000000         12.0  4.083388e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2004  NTLmaxD_2004  NTLminD_2004  NTLcfc_D_2004  \\\n",
      "11475  ...  2.430588e+38  3.402823e+38  2.538352e+00     200.857143   \n",
      "18008  ...  2.552118e+38  3.402823e+38  2.366830e+00     208.750000   \n",
      "16051  ...  3.317753e+38  3.402823e+38  2.308259e+00     250.350000   \n",
      "13384  ...  3.223727e+38  3.402823e+38  2.330779e+00     245.368421   \n",
      "22332  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "10555  ...  3.159765e+38  3.402823e+38  2.592597e+00     241.500000   \n",
      "1793   ...  2.916706e+38  3.402823e+38  2.711599e+00     227.857143   \n",
      "59     ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "15653  ...  3.093476e+38  3.402823e+38  2.405933e+00     237.545455   \n",
      "16678  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "\n",
      "       NTLctD_2005  NTLsumD_2005  NTLavgD_2005  NTLmaxD_2005  NTLminD_2005  \\\n",
      "11475          7.0  1.701412e+39  2.430588e+38  3.402823e+38  2.230212e+00   \n",
      "18008          4.0  1.020847e+39  2.552118e+38  3.402823e+38  1.877502e+00   \n",
      "16051         40.0  1.327101e+40  3.317753e+38  3.402823e+38  1.964373e+00   \n",
      "13384         19.0  6.125082e+39  3.223727e+38  3.402823e+38  2.031392e+00   \n",
      "22332         19.0  6.465365e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "10555         14.0  4.423671e+39  3.159765e+38  3.402823e+38  2.177203e+00   \n",
      "1793           7.0  2.041694e+39  2.916706e+38  3.402823e+38  1.980699e+00   \n",
      "59            20.0  6.805647e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "15653         11.0  3.402823e+39  3.093476e+38  3.402823e+38  1.999254e+00   \n",
      "16678         12.0  4.083388e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLcfc_D_2005  \n",
      "11475     201.857143  \n",
      "18008     209.750000  \n",
      "16051     250.375000  \n",
      "13384     245.000000  \n",
      "22332     255.000000  \n",
      "10555     241.500000  \n",
      "1793      228.142857  \n",
      "59        255.000000  \n",
      "15653     238.090909  \n",
      "16678     255.000000  \n",
      "\n",
      "[10 rows x 43 columns]\n",
      "Loading data for Msk_D_2006_avg.tif. Sun Feb 19 10:33:06 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2006. Sun Feb 19 10:33:07 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:33:08 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:33:08 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:33:37 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "417037  3.402823e+38   351319\n",
      "169384  3.402823e+38   359342\n",
      "41095   3.402823e+38   597335\n",
      "591327  3.402823e+38    84361\n",
      "197114  3.402823e+38   568787\n",
      "311645  3.402823e+38   359103\n",
      "88046   3.402823e+38   581592\n",
      "456205  3.402823e+38   162616\n",
      "74870   3.402823e+38   388946\n",
      "453179  3.402823e+38   358868\n",
      "\n",
      "Exported as table. Sun Feb 19 10:33:39 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:33:39 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2006. Sun Feb 19 10:33:39 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "22605   556871         30.0  9.187623e+39  3.062541e+38  3.402823e+38   \n",
      "3899     92495          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "18454   455741          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "6320    156201          1.0  3.369249e+00  3.369249e+00  3.369249e+00   \n",
      "8671    203935          1.0  2.737273e+00  2.737273e+00  2.737273e+00   \n",
      "14124   388776         22.0  7.145929e+39  3.248150e+38  3.402823e+38   \n",
      "5794    138046          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "5776    138011         55.0  1.837525e+40  3.340954e+38  3.402823e+38   \n",
      "4884    117501          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22402   528494         46.0  1.565299e+40  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "22605  2.639011e+00     235.300000         30.0  9.187623e+39  3.062541e+38   \n",
      "3899   2.460069e+00     187.666667          3.0  6.805647e+38  2.268549e+38   \n",
      "18454  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "6320   3.369249e+00      50.000000          1.0  3.709086e+00  3.709086e+00   \n",
      "8671   2.737273e+00      58.000000          1.0  2.886349e+00  2.886349e+00   \n",
      "14124  2.789682e+00     245.909091         22.0  7.145929e+39  3.248150e+38   \n",
      "5794   2.979439e+00     151.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "5776   2.896429e+00     251.309091         55.0  1.837525e+40  3.340954e+38   \n",
      "4884   3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "22402  3.402823e+38     255.000000         46.0  1.565299e+40  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumD_2005  NTLavgD_2005  NTLmaxD_2005  NTLminD_2005  \\\n",
      "22605  ...  9.187623e+39  3.062541e+38  3.402823e+38  2.025033e+00   \n",
      "3899   ...  6.805647e+38  2.268549e+38  3.402823e+38  2.200969e+00   \n",
      "18454  ...  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "6320   ...  2.566675e+00  2.566675e+00  2.566675e+00  2.566675e+00   \n",
      "8671   ...  1.931944e+00  1.931944e+00  1.931944e+00  1.931944e+00   \n",
      "14124  ...  7.145929e+39  3.248150e+38  3.402823e+38  2.026177e+00   \n",
      "5794   ...  3.402823e+38  1.701412e+38  3.402823e+38  2.039220e+00   \n",
      "5776   ...  1.837525e+40  3.340954e+38  3.402823e+38  2.202733e+00   \n",
      "4884   ...  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22402  ...  1.565299e+40  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLcfc_D_2005  NTLctD_2006  NTLsumD_2006  NTLavgD_2006  NTLmaxD_2006  \\\n",
      "22605     236.266667         30.0  9.187623e+39  3.062541e+38  3.402823e+38   \n",
      "3899      190.000000          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "18454     255.000000          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "6320       67.000000          1.0  2.493585e+00  2.493585e+00  2.493585e+00   \n",
      "8671       71.000000          1.0  2.046321e+00  2.046321e+00  2.046321e+00   \n",
      "14124     246.818182         22.0  7.145929e+39  3.248150e+38  3.402823e+38   \n",
      "5794      162.500000          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "5776      251.618182         55.0  1.837525e+40  3.340954e+38  3.402823e+38   \n",
      "4884      255.000000          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22402     255.000000         46.0  1.565299e+40  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_2006  \n",
      "22605  2.105045e+00  \n",
      "3899   2.567633e+00  \n",
      "18454  3.402823e+38  \n",
      "6320   2.493585e+00  \n",
      "8671   2.046321e+00  \n",
      "14124  2.126508e+00  \n",
      "5794   2.319892e+00  \n",
      "5776   2.456486e+00  \n",
      "4884   3.402823e+38  \n",
      "22402  3.402823e+38  \n",
      "\n",
      "[10 rows x 48 columns]\n",
      "Loading data for Msk_D_2006_cfc.tif. Sun Feb 19 10:33:41 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2006. Sun Feb 19 10:33:42 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:33:43 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:33:43 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:34:12 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "80227   255   388932\n",
      "194748  255   388882\n",
      "310332   57   203935\n",
      "35991   255   568902\n",
      "388522  255   102717\n",
      "200123  255   388882\n",
      "161514  255   400842\n",
      "83820   255   388931\n",
      "655109  255       41\n",
      "528160  255      724\n",
      "\n",
      "Exported as table. Sun Feb 19 10:34:13 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:34:13 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2006. Sun Feb 19 10:34:14 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "20121   468124          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "656       5255          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "8454    203614          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "20015   462687          1.0  2.518939e+00  2.518939e+00  2.518939e+00   \n",
      "13          14         30.0  8.507059e+39  2.835686e+38  3.402823e+38   \n",
      "7983    192799         53.0  1.769468e+40  3.338619e+38  3.402823e+38   \n",
      "8832    204326          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "12236   294334          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "11158   248080          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "12468   294888         22.0  7.145929e+39  3.248150e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "20121  2.525850e+00     155.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "656    3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "8454   3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "20015  2.518939e+00      56.000000          1.0  2.838846e+00  2.838846e+00   \n",
      "13     2.894952e+00     221.166667         30.0  8.507059e+39  2.835686e+38   \n",
      "7983   2.450550e+00     251.226415         53.0  1.769468e+40  3.338619e+38   \n",
      "8832   3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "12236  2.721320e+00     154.500000          2.0  3.402823e+38  1.701412e+38   \n",
      "11158  3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "12468  2.966434e+00     245.909091         22.0  7.145929e+39  3.248150e+38   \n",
      "\n",
      "       ...  NTLavgD_2005  NTLmaxD_2005  NTLminD_2005  NTLcfc_D_2005  \\\n",
      "20121  ...  1.701412e+38  3.402823e+38  2.010959e+00     163.500000   \n",
      "656    ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "8454   ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "20015  ...  1.973333e+00  1.973333e+00  1.973333e+00      75.000000   \n",
      "13     ...  2.835686e+38  3.402823e+38  2.123733e+00     223.733333   \n",
      "7983   ...  3.338619e+38  3.402823e+38  2.132130e+00     251.490566   \n",
      "8832   ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "12236  ...  1.701412e+38  3.402823e+38  1.930238e+00     161.500000   \n",
      "11158  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "12468  ...  3.248150e+38  3.402823e+38  1.832424e+00     246.636364   \n",
      "\n",
      "       NTLctD_2006  NTLsumD_2006  NTLavgD_2006  NTLmaxD_2006  NTLminD_2006  \\\n",
      "20121          2.0  3.402823e+38  1.701412e+38  3.402823e+38  2.197619e+00   \n",
      "656            3.0  1.020847e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "8454           1.0  3.402823e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "20015          1.0  2.089330e+00  2.089330e+00  2.089330e+00  2.089330e+00   \n",
      "13            30.0  8.507059e+39  2.835686e+38  3.402823e+38  2.546651e+00   \n",
      "7983          53.0  1.769468e+40  3.338619e+38  3.402823e+38  2.255617e+00   \n",
      "8832           4.0  1.361129e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "12236          2.0  3.402823e+38  1.701412e+38  3.402823e+38  2.179253e+00   \n",
      "11158          8.0  2.722259e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "12468         22.0  7.145929e+39  3.248150e+38  3.402823e+38  2.303636e+00   \n",
      "\n",
      "       NTLcfc_D_2006  \n",
      "20121     157.000000  \n",
      "656       255.000000  \n",
      "8454      255.000000  \n",
      "20015      58.000000  \n",
      "13        221.900000  \n",
      "7983      251.169811  \n",
      "8832      255.000000  \n",
      "12236     159.000000  \n",
      "11158     255.000000  \n",
      "12468     246.363636  \n",
      "\n",
      "[10 rows x 49 columns]\n",
      "Loading data for Msk_D_2007_avg.tif. Sun Feb 19 10:34:15 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2007. Sun Feb 19 10:34:16 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:34:17 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:34:18 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:34:46 2023 \n",
      "\n",
      "                   Z  Sett_ID\n",
      "109255  3.402823e+38   400843\n",
      "590199  3.402823e+38     2714\n",
      "341493  3.402823e+38    37253\n",
      "436897  3.402823e+38   351312\n",
      "289944  3.402823e+38   556915\n",
      "282887  3.402823e+38   359265\n",
      "99472   3.402823e+38   468653\n",
      "133494  3.402823e+38   388946\n",
      "374414  2.589366e+00   272300\n",
      "589083  3.402823e+38   351211\n",
      "\n",
      "Exported as table. Sun Feb 19 10:34:48 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:34:48 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2007. Sun Feb 19 10:34:48 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "8871    204411          1.0  2.866604e+00  2.866604e+00  2.866604e+00   \n",
      "12346   294539          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "18243   454224          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "22669   568611         28.0  9.187623e+39  3.281294e+38  3.402823e+38   \n",
      "10693   242571          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "20355   468473          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "12275   294435         11.0  3.402823e+39  3.093476e+38  3.402823e+38   \n",
      "2980     61206          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "17625   445879          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "16898   417482          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "8871   2.866604e+00      58.000000          1.0  2.951050e+00  2.951050e+00   \n",
      "12346  3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "18243  2.495152e+00     189.333333          3.0  6.805647e+38  2.268549e+38   \n",
      "22669  2.430000e+00     248.071429         28.0  9.187623e+39  3.281294e+38   \n",
      "10693  3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "20355  3.402823e+38     255.000000          6.0  2.041694e+39  3.402823e+38   \n",
      "12275  2.826535e+00     236.545455         11.0  3.402823e+39  3.093476e+38   \n",
      "2980   3.100461e+00     203.750000          4.0  1.020847e+39  2.552118e+38   \n",
      "17625  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "16898  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumD_2006  NTLavgD_2006  NTLmaxD_2006  NTLminD_2006  \\\n",
      "8871   ...  2.294179e+00  2.294179e+00  2.294179e+00  2.294179e+00   \n",
      "12346  ...  2.722259e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "18243  ...  6.805647e+38  2.268549e+38  3.402823e+38  2.218006e+00   \n",
      "22669  ...  9.187623e+39  3.281294e+38  3.402823e+38  1.999249e+00   \n",
      "10693  ...  1.701412e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "20355  ...  2.041694e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "12275  ...  3.402823e+39  3.093476e+38  3.402823e+38  2.191081e+00   \n",
      "2980   ...  1.020847e+39  2.552118e+38  3.402823e+38  2.532006e+00   \n",
      "17625  ...  6.805647e+38  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "16898  ...  1.361129e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLcfc_D_2006  NTLctD_2007  NTLsumD_2007  NTLavgD_2007  NTLmaxD_2007  \\\n",
      "8871       55.000000          1.0  2.562987e+00  2.562987e+00  2.562987e+00   \n",
      "12346     255.000000          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "18243     189.000000          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "22669     248.178571         28.0  9.187623e+39  3.281294e+38  3.402823e+38   \n",
      "10693     255.000000          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "20355     255.000000          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "12275     237.545455         11.0  3.402823e+39  3.093476e+38  3.402823e+38   \n",
      "2980      205.000000          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "17625     255.000000          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "16898     255.000000          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_2007  \n",
      "8871   2.562987e+00  \n",
      "12346  3.402823e+38  \n",
      "18243  2.524181e+00  \n",
      "22669  2.308303e+00  \n",
      "10693  3.402823e+38  \n",
      "20355  3.402823e+38  \n",
      "12275  2.511331e+00  \n",
      "2980   2.665804e+00  \n",
      "17625  3.402823e+38  \n",
      "16898  3.402823e+38  \n",
      "\n",
      "[10 rows x 54 columns]\n",
      "Loading data for Msk_D_2007_cfc.tif. Sun Feb 19 10:34:49 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2007. Sun Feb 19 10:34:50 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:34:51 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:34:51 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:20 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "580640  255    84362\n",
      "81464   255   468653\n",
      "483366  255   165792\n",
      "66893   255   458333\n",
      "330742  255    51192\n",
      "460492  255    40687\n",
      "383436   61   271513\n",
      "176255  255   509130\n",
      "291178  255   388579\n",
      "31735   255   458333\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:21 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:21 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2007. Sun Feb 19 10:35:21 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "17392   445511          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "9965    227144          5.0  1.361129e+39  2.722259e+38  3.402823e+38   \n",
      "18270   454253          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "19337   457936          NaN           NaN           NaN           NaN   \n",
      "1966     38637          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "18865   457347          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "171        204          7.0  2.381976e+39  3.402823e+38  3.402823e+38   \n",
      "4733    117300          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "14908   400795         56.0  1.871553e+40  3.342059e+38  3.402823e+38   \n",
      "14207   388871         20.0  6.465365e+39  3.232682e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "17392  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "9965   2.819119e+00     215.400000          5.0  1.361129e+39  2.722259e+38   \n",
      "18270  2.524063e+00     205.500000          4.0  1.020847e+39  2.552118e+38   \n",
      "19337           NaN            NaN          NaN           NaN           NaN   \n",
      "1966   3.155841e+00     221.166667          6.0  1.701412e+39  2.835686e+38   \n",
      "18865  2.528473e+00     188.666667          3.0  6.805647e+38  2.268549e+38   \n",
      "171    3.402823e+38     255.000000          7.0  2.381976e+39  3.402823e+38   \n",
      "4733   2.889306e+00     188.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "14908  2.776423e+00     251.321429         56.0  1.871553e+40  3.342059e+38   \n",
      "14207  2.533333e+00     245.100000         20.0  6.465365e+39  3.232682e+38   \n",
      "\n",
      "       ...  NTLavgD_2006  NTLmaxD_2006  NTLminD_2006  NTLcfc_D_2006  \\\n",
      "17392  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "9965   ...  2.722259e+38  3.402823e+38  2.386364e+00     215.000000   \n",
      "18270  ...  2.552118e+38  3.402823e+38  2.230896e+00     205.250000   \n",
      "19337  ...           NaN           NaN           NaN            NaN   \n",
      "1966   ...  2.835686e+38  3.402823e+38  2.018432e+00     220.833333   \n",
      "18865  ...  2.268549e+38  3.402823e+38  2.077936e+00     189.666667   \n",
      "171    ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "4733   ...  2.268549e+38  3.402823e+38  2.080823e+00     187.333333   \n",
      "14908  ...  3.342059e+38  3.402823e+38  2.229192e+00     251.482143   \n",
      "14207  ...  3.232682e+38  3.402823e+38  2.041270e+00     245.250000   \n",
      "\n",
      "       NTLctD_2007  NTLsumD_2007  NTLavgD_2007  NTLmaxD_2007  NTLminD_2007  \\\n",
      "17392          4.0  1.361129e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "9965           5.0  1.361129e+39  2.722259e+38  3.402823e+38  2.595960e+00   \n",
      "18270          4.0  1.020847e+39  2.552118e+38  3.402823e+38  2.493243e+00   \n",
      "19337          NaN           NaN           NaN           NaN           NaN   \n",
      "1966           6.0  1.701412e+39  2.835686e+38  3.402823e+38  2.507292e+00   \n",
      "18865          3.0  6.805647e+38  2.268549e+38  3.402823e+38  2.301299e+00   \n",
      "171            7.0  2.381976e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4733           3.0  6.805647e+38  2.268549e+38  3.402823e+38  2.713663e+00   \n",
      "14908         56.0  1.871553e+40  3.342059e+38  3.402823e+38  2.470123e+00   \n",
      "14207         20.0  6.465365e+39  3.232682e+38  3.402823e+38  2.523507e+00   \n",
      "\n",
      "       NTLcfc_D_2007  \n",
      "17392     255.000000  \n",
      "9965      216.600000  \n",
      "18270     208.000000  \n",
      "19337            NaN  \n",
      "1966      221.833333  \n",
      "18865     192.000000  \n",
      "171       255.000000  \n",
      "4733      188.000000  \n",
      "14908     251.625000  \n",
      "14207     245.550000  \n",
      "\n",
      "[10 rows x 55 columns]\n",
      "Loading data for Msk_D_2008_avg.tif. Sun Feb 19 10:35:22 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2008. Sun Feb 19 10:35:23 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:24 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:24 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:25 2023 \n",
      "\n",
      "               Z  Sett_ID\n",
      "439120  2.958333   165769\n",
      "214921  2.457143   445743\n",
      "267272  2.891892   514917\n",
      "386675  2.957747    51940\n",
      "73620   2.712500   597211\n",
      "310540  2.986667   310166\n",
      "530816  2.813559      604\n",
      "296899  3.454545   501561\n",
      "537964  2.666667      214\n",
      "358371  2.558442   318745\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:25 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:25 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2008. Sun Feb 19 10:35:25 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "15587   409423         51.0  1.735440e+40  3.402823e+38  3.402823e+38   \n",
      "336        390          1.0  4.290984e+00  4.290984e+00  4.290984e+00   \n",
      "6846    166130          9.0  3.062541e+39  3.402823e+38  3.402823e+38   \n",
      "6792    166027         20.0  6.465365e+39  3.232682e+38  3.402823e+38   \n",
      "3703     87385          9.0  2.722259e+39  3.024732e+38  3.402823e+38   \n",
      "21329   501528         39.0  1.327101e+40  3.402823e+38  3.402823e+38   \n",
      "22505   528882         14.0  4.423671e+39  3.159765e+38  3.402823e+38   \n",
      "15392   409151          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "4971    117637         10.0  3.062541e+39  3.062541e+38  3.402823e+38   \n",
      "5806    138062         55.0  1.837525e+40  3.340954e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "15587  3.402823e+38     255.000000         51.0  1.735440e+40  3.402823e+38   \n",
      "336    4.290984e+00      50.000000          1.0  4.544175e+00  4.544175e+00   \n",
      "6846   3.402823e+38     255.000000          9.0  3.062541e+39  3.402823e+38   \n",
      "6792   2.565055e+00     245.200000         20.0  6.465365e+39  3.232682e+38   \n",
      "3703   2.731517e+00     233.000000          9.0  2.722259e+39  3.024732e+38   \n",
      "21329  3.402823e+38     255.000000         39.0  1.327101e+40  3.402823e+38   \n",
      "22505  2.445833e+00     240.642857         14.0  4.423671e+39  3.159765e+38   \n",
      "15392  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "4971   2.884146e+00     235.100000         10.0  3.062541e+39  3.062541e+38   \n",
      "5806   2.941802e+00     251.272727         55.0  1.837525e+40  3.340954e+38   \n",
      "\n",
      "       ...  NTLsumD_2007  NTLavgD_2007  NTLmaxD_2007  NTLminD_2007  \\\n",
      "15587  ...  1.735440e+40  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "336    ...  5.932918e+00  5.932918e+00  5.932918e+00  5.932918e+00   \n",
      "6846   ...  3.062541e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "6792   ...  6.465365e+39  3.232682e+38  3.402823e+38  3.045815e+00   \n",
      "3703   ...  2.722259e+39  3.024732e+38  3.402823e+38  2.376579e+00   \n",
      "21329  ...  1.327101e+40  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "22505  ...  4.423671e+39  3.159765e+38  3.402823e+38  2.359030e+00   \n",
      "15392  ...  1.361129e+39  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "4971   ...  3.062541e+39  3.062541e+38  3.402823e+38  2.545804e+00   \n",
      "5806   ...  1.837525e+40  3.340954e+38  3.402823e+38  2.725914e+00   \n",
      "\n",
      "       NTLcfc_D_2007  NTLctD_2008  NTLsumD_2008  NTLavgD_2008  NTLmaxD_2008  \\\n",
      "15587     255.000000          NaN           NaN           NaN           NaN   \n",
      "336        59.000000          1.0      4.538462      4.538462      4.538462   \n",
      "6846      255.000000          NaN           NaN           NaN           NaN   \n",
      "6792      245.450000          1.0      4.338710      4.338710      4.338710   \n",
      "3703      233.777778          1.0      2.736111      2.736111      2.736111   \n",
      "21329     255.000000          NaN           NaN           NaN           NaN   \n",
      "22505     241.928571          1.0      3.075949      3.075949      3.075949   \n",
      "15392     255.000000          NaN           NaN           NaN           NaN   \n",
      "4971      235.700000          1.0      2.878788      2.878788      2.878788   \n",
      "5806      251.436364          1.0      3.054795      3.054795      3.054795   \n",
      "\n",
      "       NTLminD_2008  \n",
      "15587           NaN  \n",
      "336        4.538462  \n",
      "6846            NaN  \n",
      "6792       4.338710  \n",
      "3703       2.736111  \n",
      "21329           NaN  \n",
      "22505      3.075949  \n",
      "15392           NaN  \n",
      "4971       2.878788  \n",
      "5806       3.054795  \n",
      "\n",
      "[10 rows x 60 columns]\n",
      "Loading data for Msk_D_2008_cfc.tif. Sun Feb 19 10:35:27 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2008. Sun Feb 19 10:35:28 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:29 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:29 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:30 2023 \n",
      "\n",
      "         Z  Sett_ID\n",
      "539787  62    12352\n",
      "256398  73   415483\n",
      "296557  68   117617\n",
      "352499  80   142438\n",
      "411175  66   129803\n",
      "419988  66    51731\n",
      "221804  77   388848\n",
      "370431  70    51989\n",
      "447021  68    52047\n",
      "513659  63        8\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:30 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:30 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2008. Sun Feb 19 10:35:30 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "18258   454241         23.0  7.486212e+39  3.254875e+38  3.402823e+38   \n",
      "15099   404552         12.0  4.083388e+39  3.402823e+38  3.402823e+38   \n",
      "18256   454239          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "8703    203994          5.0  1.020847e+39  2.041694e+38  3.402823e+38   \n",
      "11364   265702          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "10600   236696         56.0  1.871553e+40  3.342059e+38  3.402823e+38   \n",
      "17637   445899          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "1489     35167          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "20153   468185          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "2740     51956          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "18258  2.523142e+00     246.260870         23.0  7.486212e+39  3.254875e+38   \n",
      "15099  3.402823e+38     255.000000         12.0  4.083388e+39  3.402823e+38   \n",
      "18256  2.661770e+00     205.000000          4.0  1.020847e+39  2.552118e+38   \n",
      "8703   2.681389e+00     177.400000          5.0  1.020847e+39  2.041694e+38   \n",
      "11364  2.817045e+00     188.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "10600  2.665139e+00     251.464286         56.0  1.871553e+40  3.342059e+38   \n",
      "17637  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "1489   3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "20153  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "2740   3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2007  NTLmaxD_2007  NTLminD_2007  NTLcfc_D_2007  \\\n",
      "18258  ...  3.254875e+38  3.402823e+38  2.424962e+00     246.739130   \n",
      "15099  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "18256  ...  2.552118e+38  3.402823e+38  2.514118e+00     207.000000   \n",
      "8703   ...  2.041694e+38  3.402823e+38  2.541177e+00     177.600000   \n",
      "11364  ...  2.268549e+38  3.402823e+38  2.333810e+00     190.666667   \n",
      "10600  ...  3.342059e+38  3.402823e+38  2.762792e+00     251.535714   \n",
      "17637  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "1489   ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "20153  ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "2740   ...  3.402823e+38  3.402823e+38  3.402823e+38     255.000000   \n",
      "\n",
      "       NTLctD_2008  NTLsumD_2008  NTLavgD_2008  NTLmaxD_2008  NTLminD_2008  \\\n",
      "18258          1.0      2.800000      2.800000      2.800000      2.800000   \n",
      "15099          NaN           NaN           NaN           NaN           NaN   \n",
      "18256          1.0      2.746032      2.746032      2.746032      2.746032   \n",
      "8703           2.0      6.001093      3.000546      3.066667      2.934426   \n",
      "11364          1.0      2.707692      2.707692      2.707692      2.707692   \n",
      "10600          1.0      2.476191      2.476191      2.476191      2.476191   \n",
      "17637          NaN           NaN           NaN           NaN           NaN   \n",
      "1489           NaN           NaN           NaN           NaN           NaN   \n",
      "20153          NaN           NaN           NaN           NaN           NaN   \n",
      "2740           NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_D_2008  \n",
      "18258           70.0  \n",
      "15099            NaN  \n",
      "18256           63.0  \n",
      "8703            61.0  \n",
      "11364           66.0  \n",
      "10600           65.0  \n",
      "17637            NaN  \n",
      "1489             NaN  \n",
      "20153            NaN  \n",
      "2740             NaN  \n",
      "\n",
      "[10 rows x 61 columns]\n",
      "Loading data for Msk_D_2009_avg.tif. Sun Feb 19 10:35:31 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2009. Sun Feb 19 10:35:32 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:33 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:33 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:34 2023 \n",
      "\n",
      "                Z  Sett_ID\n",
      "454977   2.633333   165870\n",
      "465051   3.120000   236602\n",
      "308056  10.913043   227047\n",
      "216631   2.192308   388866\n",
      "165428   3.138889   458144\n",
      "345991   2.724138   294261\n",
      "355987   3.166667   309658\n",
      "307256   2.636364   295080\n",
      "389409   3.060606   277878\n",
      "219638   2.785714   445238\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:34 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:34 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2009. Sun Feb 19 10:35:34 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "11552   266102         14.0  4.423671e+39  3.159765e+38  3.402823e+38   \n",
      "10423   231847          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "3086     66081          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "1177     29965          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "14921   400814         23.0  7.826494e+39  3.402823e+38  3.402823e+38   \n",
      "4120     97738          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "21154   482476          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "23355   597291         13.0  4.083388e+39  3.141068e+38  3.402823e+38   \n",
      "4528    102804         27.0  8.507059e+39  3.150762e+38  3.402823e+38   \n",
      "4458    102709         10.0  3.402823e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "11552  3.238290e+00     241.071429         14.0  4.423671e+39  3.159765e+38   \n",
      "10423  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "3086   3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "1177   3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "14921  3.402823e+38     255.000000         23.0  7.826494e+39  3.402823e+38   \n",
      "4120   3.402823e+38     255.000000          6.0  2.041694e+39  3.402823e+38   \n",
      "21154  2.482906e+00     226.428571          7.0  2.041694e+39  2.916706e+38   \n",
      "23355  2.612825e+00     239.692308         13.0  4.083388e+39  3.141068e+38   \n",
      "4528   2.654396e+00     240.074074         27.0  8.507059e+39  3.150762e+38   \n",
      "4458   3.402823e+38     255.000000         10.0  3.402823e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumD_2008  NTLavgD_2008  NTLmaxD_2008  NTLminD_2008  \\\n",
      "11552  ...      2.621212      2.621212      2.621212      2.621212   \n",
      "10423  ...           NaN           NaN           NaN           NaN   \n",
      "3086   ...           NaN           NaN           NaN           NaN   \n",
      "1177   ...           NaN           NaN           NaN           NaN   \n",
      "14921  ...           NaN           NaN           NaN           NaN   \n",
      "4120   ...           NaN           NaN           NaN           NaN   \n",
      "21154  ...      2.750000      2.750000      2.750000      2.750000   \n",
      "23355  ...      2.717949      2.717949      2.717949      2.717949   \n",
      "4528   ...      6.161106      3.080553      3.205882      2.955224   \n",
      "4458   ...           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_D_2008  NTLctD_2009  NTLsumD_2009  NTLavgD_2009  NTLmaxD_2009  \\\n",
      "11552           68.0          NaN           NaN           NaN           NaN   \n",
      "10423            NaN          1.0      3.448276      3.448276      3.448276   \n",
      "3086             NaN          NaN           NaN           NaN           NaN   \n",
      "1177             NaN          1.0      3.250000      3.250000      3.250000   \n",
      "14921            NaN          1.0      2.933333      2.933333      2.933333   \n",
      "4120             NaN          NaN           NaN           NaN           NaN   \n",
      "21154           68.0          NaN           NaN           NaN           NaN   \n",
      "23355           78.0          NaN           NaN           NaN           NaN   \n",
      "4528            68.5          2.0      6.653846      3.326923      3.346154   \n",
      "4458             NaN          1.0      3.750000      3.750000      3.750000   \n",
      "\n",
      "       NTLminD_2009  \n",
      "11552           NaN  \n",
      "10423      3.448276  \n",
      "3086            NaN  \n",
      "1177       3.250000  \n",
      "14921      2.933333  \n",
      "4120            NaN  \n",
      "21154           NaN  \n",
      "23355           NaN  \n",
      "4528       3.307692  \n",
      "4458       3.750000  \n",
      "\n",
      "[10 rows x 66 columns]\n",
      "Loading data for Msk_D_2009_cfc.tif. Sun Feb 19 10:35:36 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2009. Sun Feb 19 10:35:37 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:38 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:38 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:39 2023 \n",
      "\n",
      "         Z  Sett_ID\n",
      "360000  31   221685\n",
      "356488  33   309649\n",
      "99632   25   475102\n",
      "193240  26   445428\n",
      "286947  31   404486\n",
      "534412  31    21524\n",
      "239423  29   528471\n",
      "289514  30   396411\n",
      "261660  27   409133\n",
      "234693  30   409739\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:39 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:39 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2009. Sun Feb 19 10:35:39 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "23064   581325          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "14158   388817         23.0  7.486212e+39  3.254875e+38  3.402823e+38   \n",
      "4144     98138         18.0  5.784800e+39  3.213778e+38  3.402823e+38   \n",
      "4839    117427          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "13137   311272          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "21160   483068         17.0  5.784800e+39  3.402823e+38  3.402823e+38   \n",
      "11234   250016         13.0  4.423671e+39  3.402823e+38  3.402823e+38   \n",
      "4419    102661          7.0  2.381976e+39  3.402823e+38  3.402823e+38   \n",
      "8179    203147          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "4843    117433         19.0  6.125082e+39  3.223727e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "23064  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "14158  2.827273e+00     246.304348         23.0  7.486212e+39  3.254875e+38   \n",
      "4144   2.560729e+00     243.833333         18.0  5.784800e+39  3.213778e+38   \n",
      "4839   3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "13137  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "21160  3.402823e+38     255.000000         17.0  5.784800e+39  3.402823e+38   \n",
      "11234  3.402823e+38     255.000000         13.0  4.423671e+39  3.402823e+38   \n",
      "4419   3.402823e+38     255.000000          7.0  2.381976e+39  3.402823e+38   \n",
      "8179   2.886364e+00     204.500000          4.0  1.020847e+39  2.552118e+38   \n",
      "4843   3.067460e+00     244.421053         19.0  6.125082e+39  3.223727e+38   \n",
      "\n",
      "       ...  NTLavgD_2008  NTLmaxD_2008  NTLminD_2008  NTLcfc_D_2008  \\\n",
      "23064  ...           NaN           NaN           NaN            NaN   \n",
      "14158  ...      3.526316      3.526316      3.526316           76.0   \n",
      "4144   ...      3.029851      3.029851      3.029851           69.0   \n",
      "4839   ...           NaN           NaN           NaN            NaN   \n",
      "13137  ...           NaN           NaN           NaN            NaN   \n",
      "21160  ...           NaN           NaN           NaN            NaN   \n",
      "11234  ...           NaN           NaN           NaN            NaN   \n",
      "4419   ...           NaN           NaN           NaN            NaN   \n",
      "8179   ...      2.893333      2.893333      2.893333           75.0   \n",
      "4843   ...      2.774648      2.774648      2.774648           71.0   \n",
      "\n",
      "       NTLctD_2009  NTLsumD_2009  NTLavgD_2009  NTLmaxD_2009  NTLminD_2009  \\\n",
      "23064          1.0      3.000000      3.000000      3.000000      3.000000   \n",
      "14158          1.0      3.480000      3.480000      3.480000      3.480000   \n",
      "4144           NaN           NaN           NaN           NaN           NaN   \n",
      "4839           1.0      2.782609      2.782609      2.782609      2.782609   \n",
      "13137          NaN           NaN           NaN           NaN           NaN   \n",
      "21160          1.0      2.275862      2.275862      2.275862      2.275862   \n",
      "11234          1.0      2.962963      2.962963      2.962963      2.962963   \n",
      "4419           NaN           NaN           NaN           NaN           NaN   \n",
      "8179           NaN           NaN           NaN           NaN           NaN   \n",
      "4843           NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_D_2009  \n",
      "23064            NaN  \n",
      "14158           26.0  \n",
      "4144            28.0  \n",
      "4839             NaN  \n",
      "13137            NaN  \n",
      "21160            NaN  \n",
      "11234            NaN  \n",
      "4419             NaN  \n",
      "8179            33.0  \n",
      "4843            23.0  \n",
      "\n",
      "[10 rows x 67 columns]\n",
      "Loading data for Msk_D_2010_avg.tif. Sun Feb 19 10:35:41 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2010. Sun Feb 19 10:35:41 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:42 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:42 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:43 2023 \n",
      "\n",
      "               Z  Sett_ID\n",
      "224933  3.830986   400594\n",
      "170039  3.691176   445953\n",
      "319539  4.060606   138190\n",
      "186386  3.788732   509105\n",
      "409262  3.926471   277738\n",
      "375807  7.014084   247086\n",
      "419904  4.687500   236740\n",
      "306908  4.688525   138045\n",
      "329688  5.478261   227036\n",
      "308843  4.454545   203971\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:43 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:43 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2010. Sun Feb 19 10:35:44 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "13537   319236        102.0  3.470880e+40  3.402823e+38  3.402823e+38   \n",
      "8726    204048          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "18231   454211          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "17759   446872          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "10922   247605          1.0  2.622596e+00  2.622596e+00  2.622596e+00   \n",
      "17641   445909          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "2394     51495         18.0  5.444518e+39  3.024732e+38  3.402823e+38   \n",
      "11434   265853          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "14306   391477         14.0  4.763953e+39  3.402823e+38  3.402823e+38   \n",
      "10147   227426          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "13537  3.402823e+38     255.000000        102.0  3.470880e+40  3.402823e+38   \n",
      "8726   3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "18231  2.765233e+00     154.500000          2.0  3.402823e+38  1.701412e+38   \n",
      "17759  3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "10922  2.622596e+00      59.000000          1.0  2.793933e+00  2.793933e+00   \n",
      "17641  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "2394   2.813397e+00     232.666667         18.0  5.444518e+39  3.024732e+38   \n",
      "11434  3.005095e+00     190.333333          3.0  6.805647e+38  2.268549e+38   \n",
      "14306  3.402823e+38     255.000000         14.0  4.763953e+39  3.402823e+38   \n",
      "10147  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumD_2009  NTLavgD_2009  NTLmaxD_2009  NTLminD_2009  \\\n",
      "13537  ...      3.000000      3.000000      3.000000      3.000000   \n",
      "8726   ...      3.000000      3.000000      3.000000      3.000000   \n",
      "18231  ...      2.833333      2.833333      2.833333      2.833333   \n",
      "17759  ...      3.115385      3.115385      3.115385      3.115385   \n",
      "10922  ...           NaN           NaN           NaN           NaN   \n",
      "17641  ...      2.925926      2.925926      2.925926      2.925926   \n",
      "2394   ...      7.901806      2.633935      2.785714      2.482759   \n",
      "11434  ...      2.666667      2.666667      2.666667      2.666667   \n",
      "14306  ...           NaN           NaN           NaN           NaN   \n",
      "10147  ...      6.642857      3.321429      3.357143      3.285714   \n",
      "\n",
      "       NTLcfc_D_2009  NTLctD_2010  NTLsumD_2010  NTLavgD_2010  NTLmaxD_2010  \\\n",
      "13537            NaN          1.0      3.936508      3.936508      3.936508   \n",
      "8726             NaN          1.0      4.028572      4.028572      4.028572   \n",
      "18231           30.0          1.0      3.706667      3.706667      3.706667   \n",
      "17759            NaN          1.0      4.164383      4.164383      4.164383   \n",
      "10922           32.0          NaN           NaN           NaN           NaN   \n",
      "17641            NaN          1.0      3.432836      3.432836      3.432836   \n",
      "2394            30.0          3.0     13.628572      4.542857      4.676923   \n",
      "11434           29.0          1.0      4.708333      4.708333      4.708333   \n",
      "14306            NaN          NaN           NaN           NaN           NaN   \n",
      "10147            NaN          2.0      8.848767      4.424384      4.520000   \n",
      "\n",
      "       NTLminD_2010  \n",
      "13537      3.936508  \n",
      "8726       4.028572  \n",
      "18231      3.706667  \n",
      "17759      4.164383  \n",
      "10922           NaN  \n",
      "17641      3.432836  \n",
      "2394       4.428571  \n",
      "11434      4.708333  \n",
      "14306           NaN  \n",
      "10147      4.328767  \n",
      "\n",
      "[10 rows x 72 columns]\n",
      "Loading data for Msk_D_2010_cfc.tif. Sun Feb 19 10:35:45 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2010. Sun Feb 19 10:35:46 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:47 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:47 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:48 2023 \n",
      "\n",
      "         Z  Sett_ID\n",
      "168021  75   445574\n",
      "422214  76   265695\n",
      "173469  71   445924\n",
      "394866  65   102687\n",
      "116864  74   580905\n",
      "251975  75   508975\n",
      "441595  68    51341\n",
      "433957  77   247417\n",
      "166241  69   445138\n",
      "506546  76    51243\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:48 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:48 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2010. Sun Feb 19 10:35:48 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "18772   457226          7.0  2.381976e+39  3.402823e+38  3.402823e+38   \n",
      "6562    164681          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "13070   310210         11.0  3.062541e+39  2.784128e+38  3.402823e+38   \n",
      "6525    163494          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "1416     31243          1.0  2.663194e+00  2.663194e+00  2.663194e+00   \n",
      "17134   445165          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "10464   234047          NaN           NaN           NaN           NaN   \n",
      "5379    129905         19.0  6.125082e+39  3.223727e+38  3.402823e+38   \n",
      "3687     87361          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "18450   455597          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "18772  3.402823e+38     255.000000          7.0  2.381976e+39  3.402823e+38   \n",
      "6562   3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "13070  2.537118e+00     218.454545         11.0  3.062541e+39  2.784128e+38   \n",
      "6525   2.796526e+00     205.000000          4.0  1.020847e+39  2.552118e+38   \n",
      "1416   2.663194e+00      51.000000          1.0  2.921052e+00  2.921052e+00   \n",
      "17134  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "10464           NaN            NaN          NaN           NaN           NaN   \n",
      "5379   2.899814e+00     244.315789         19.0  6.125082e+39  3.223727e+38   \n",
      "3687   3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "18450  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2009  NTLmaxD_2009  NTLminD_2009  NTLcfc_D_2009  \\\n",
      "18772  ...      3.071429      3.071429      3.071429            NaN   \n",
      "6562   ...      3.107143      3.107143      3.107143            NaN   \n",
      "13070  ...      3.033908      3.034483      3.033333           29.0   \n",
      "6525   ...           NaN           NaN           NaN           27.0   \n",
      "1416   ...           NaN           NaN           NaN           35.0   \n",
      "17134  ...           NaN           NaN           NaN            NaN   \n",
      "10464  ...           NaN           NaN           NaN            NaN   \n",
      "5379   ...           NaN           NaN           NaN           28.0   \n",
      "3687   ...      2.758621      2.758621      2.758621            NaN   \n",
      "18450  ...      2.896552      2.896552      2.896552            NaN   \n",
      "\n",
      "       NTLctD_2010  NTLsumD_2010  NTLavgD_2010  NTLmaxD_2010  NTLminD_2010  \\\n",
      "18772          1.0      3.565217      3.565217      3.565217      3.565217   \n",
      "6562           1.0      4.080645      4.080645      4.080645      4.080645   \n",
      "13070          2.0      8.015385      4.007692      4.015385      4.000000   \n",
      "6525           NaN           NaN           NaN           NaN           NaN   \n",
      "1416           NaN           NaN           NaN           NaN           NaN   \n",
      "17134          NaN           NaN           NaN           NaN           NaN   \n",
      "10464          NaN           NaN           NaN           NaN           NaN   \n",
      "5379           NaN           NaN           NaN           NaN           NaN   \n",
      "3687           1.0      4.268657      4.268657      4.268657      4.268657   \n",
      "18450          1.0      3.743590      3.743590      3.743590      3.743590   \n",
      "\n",
      "       NTLcfc_D_2010  \n",
      "18772            NaN  \n",
      "6562             NaN  \n",
      "13070           67.5  \n",
      "6525            72.0  \n",
      "1416            70.0  \n",
      "17134            NaN  \n",
      "10464            NaN  \n",
      "5379            63.0  \n",
      "3687             NaN  \n",
      "18450            NaN  \n",
      "\n",
      "[10 rows x 73 columns]\n",
      "Loading data for Msk_D_2011_avg.tif. Sun Feb 19 10:35:50 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2011. Sun Feb 19 10:35:51 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:52 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:52 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:53 2023 \n",
      "\n",
      "        Z  Sett_ID\n",
      "351815  5   215262\n",
      "333048  3   294504\n",
      "253702  3   415892\n",
      "221010  4   400628\n",
      "337387  4   203523\n",
      "211075  4   400701\n",
      "273298  4   396490\n",
      "75389   3   597381\n",
      "232647  3   388797\n",
      "380386  3    51965\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:53 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:53 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2011. Sun Feb 19 10:35:53 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "19619   458287          NaN           NaN           NaN           NaN   \n",
      "1740     37034         24.0  7.826494e+39  3.261039e+38  3.402823e+38   \n",
      "10680   240411         96.0  3.266711e+40  3.402823e+38  3.402823e+38   \n",
      "13135   311165          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "11401   265771          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "2994     61830          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "14963   401744         15.0  4.763953e+39  3.175969e+38  3.402823e+38   \n",
      "14835   400700         30.0  9.527906e+39  3.175969e+38  3.402823e+38   \n",
      "14383   396347         13.0  4.423671e+39  3.402823e+38  3.402823e+38   \n",
      "15784   409725          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "19619           NaN            NaN          NaN           NaN           NaN   \n",
      "1740   3.463187e+00     246.375000         24.0  7.826494e+39  3.261039e+38   \n",
      "10680  3.402823e+38     255.000000         96.0  3.266711e+40  3.402823e+38   \n",
      "13135  2.797222e+00     226.142857          7.0  2.041694e+39  2.916706e+38   \n",
      "11401  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "2994   3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "14963  2.891540e+00     241.466667         15.0  4.763953e+39  3.175969e+38   \n",
      "14835  2.746846e+00     241.466667         30.0  9.527906e+39  3.175969e+38   \n",
      "14383  3.402823e+38     255.000000         13.0  4.423671e+39  3.402823e+38   \n",
      "15784  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumD_2010  NTLavgD_2010  NTLmaxD_2010  NTLminD_2010  \\\n",
      "19619  ...           NaN           NaN           NaN           NaN   \n",
      "1740   ...           NaN           NaN           NaN           NaN   \n",
      "10680  ...      4.619048      4.619048      4.619048      4.619048   \n",
      "13135  ...           NaN           NaN           NaN           NaN   \n",
      "11401  ...           NaN           NaN           NaN           NaN   \n",
      "2994   ...           NaN           NaN           NaN           NaN   \n",
      "14963  ...           NaN           NaN           NaN           NaN   \n",
      "14835  ...     12.440336      4.146779      4.442857      3.911765   \n",
      "14383  ...      4.076923      4.076923      4.076923      4.076923   \n",
      "15784  ...      3.934211      3.934211      3.934211      3.934211   \n",
      "\n",
      "       NTLcfc_D_2010  NTLctD_2011  NTLsumD_2011  NTLavgD_2011  NTLmaxD_2011  \\\n",
      "19619            NaN          NaN           NaN           NaN           NaN   \n",
      "1740            62.0          1.0           5.0           5.0           5.0   \n",
      "10680            NaN          NaN           NaN           NaN           NaN   \n",
      "13135           66.0          1.0           4.0           4.0           4.0   \n",
      "11401            NaN          NaN           NaN           NaN           NaN   \n",
      "2994             NaN          NaN           NaN           NaN           NaN   \n",
      "14963           71.0          1.0           3.0           3.0           3.0   \n",
      "14835           71.5          2.0           8.0           4.0           4.0   \n",
      "14383            NaN          NaN           NaN           NaN           NaN   \n",
      "15784            NaN          NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLminD_2011  \n",
      "19619           NaN  \n",
      "1740            5.0  \n",
      "10680           NaN  \n",
      "13135           4.0  \n",
      "11401           NaN  \n",
      "2994            NaN  \n",
      "14963           3.0  \n",
      "14835           4.0  \n",
      "14383           NaN  \n",
      "15784           NaN  \n",
      "\n",
      "[10 rows x 78 columns]\n",
      "Loading data for Msk_D_2011_cfc.tif. Sun Feb 19 10:35:55 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2011. Sun Feb 19 10:35:56 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:35:57 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:35:57 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:35:58 2023 \n",
      "\n",
      "         Z  Sett_ID\n",
      "291487  70   227657\n",
      "355483  69   215636\n",
      "202268  73   445346\n",
      "507354  63      402\n",
      "134586  76   458292\n",
      "259797  78   399848\n",
      "323918  70   227047\n",
      "240995  73   409541\n",
      "519000  59    30093\n",
      "255545  79   508756\n",
      "\n",
      "Exported as table. Sun Feb 19 10:35:58 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:35:58 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2011. Sun Feb 19 10:35:58 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "11781   277298          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "16784   417235         34.0  1.122932e+40  3.302740e+38  3.402823e+38   \n",
      "13241   315634          5.0  1.361129e+39  2.722259e+38  3.402823e+38   \n",
      "13008   310120          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "4138     98126         37.0  1.225016e+40  3.310855e+38  3.402823e+38   \n",
      "8208    203196          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "5974    146403          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "15844   409806          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "10722   247086         29.0  2.381976e+39  8.213712e+37  3.402823e+38   \n",
      "15158   404731         10.0  3.402823e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "11781  3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "16784  2.564980e+00     249.147059         34.0  1.122932e+40  3.302740e+38   \n",
      "13241  2.509818e+00     214.800000          5.0  1.361129e+39  2.722259e+38   \n",
      "13008  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "4138   2.713166e+00     249.378378         37.0  1.225016e+40  3.310855e+38   \n",
      "8208   3.154215e+00     187.666667          3.0  6.805647e+38  2.268549e+38   \n",
      "5974   3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "15844  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "10722  2.706864e+00     103.448276         29.0  2.381976e+39  8.213712e+37   \n",
      "15158  3.402823e+38     255.000000         10.0  3.402823e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2010  NTLmaxD_2010  NTLminD_2010  NTLcfc_D_2010  \\\n",
      "11781  ...      4.359729      4.411765      4.307693            NaN   \n",
      "16784  ...      3.880597      3.880597      3.880597      67.000000   \n",
      "13241  ...           NaN           NaN           NaN      71.000000   \n",
      "13008  ...      4.149254      4.149254      4.149254            NaN   \n",
      "4138   ...      4.093750      4.093750      4.093750      73.000000   \n",
      "8208   ...           NaN           NaN           NaN      72.000000   \n",
      "5974   ...           NaN           NaN           NaN            NaN   \n",
      "15844  ...      3.680000      3.680000      3.680000            NaN   \n",
      "10722  ...      6.014768      7.428571      4.671429      70.545455   \n",
      "15158  ...      3.797297      3.797297      3.797297            NaN   \n",
      "\n",
      "       NTLctD_2011  NTLsumD_2011  NTLavgD_2011  NTLmaxD_2011  NTLminD_2011  \\\n",
      "11781          NaN           NaN           NaN           NaN           NaN   \n",
      "16784          1.0           3.0      3.000000           3.0           3.0   \n",
      "13241          1.0           3.0      3.000000           3.0           3.0   \n",
      "13008          NaN           NaN           NaN           NaN           NaN   \n",
      "4138           1.0           6.0      6.000000           6.0           6.0   \n",
      "8208           1.0           4.0      4.000000           4.0           4.0   \n",
      "5974           NaN           NaN           NaN           NaN           NaN   \n",
      "15844          NaN           NaN           NaN           NaN           NaN   \n",
      "10722         22.0         113.0      5.136364           6.0           4.0   \n",
      "15158          NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_D_2011  \n",
      "11781            NaN  \n",
      "16784      75.000000  \n",
      "13241      81.000000  \n",
      "13008            NaN  \n",
      "4138       63.000000  \n",
      "8208       67.000000  \n",
      "5974             NaN  \n",
      "15844            NaN  \n",
      "10722      79.318182  \n",
      "15158            NaN  \n",
      "\n",
      "[10 rows x 79 columns]\n",
      "Loading data for Msk_D_2012_avg.tif. Sun Feb 19 10:36:00 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2012. Sun Feb 19 10:36:00 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:36:01 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:36:01 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:36:02 2023 \n",
      "\n",
      "                Z  Sett_ID\n",
      "317738   3.940299   310073\n",
      "302299  49.268658   227047\n",
      "293065   5.150000   137699\n",
      "237467   3.546667   416057\n",
      "474220   4.176471   155989\n",
      "500997   4.151515    30161\n",
      "324593   3.842857   138055\n",
      "295012   3.535211   204180\n",
      "222931   3.442857   453911\n",
      "281579   4.028986   501760\n",
      "\n",
      "Exported as table. Sun Feb 19 10:36:02 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:36:02 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2012. Sun Feb 19 10:36:03 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "22161   525266          9.0  3.062541e+39  3.402823e+38  3.402823e+38   \n",
      "8473    203649          NaN           NaN           NaN           NaN   \n",
      "12017   277793          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "8922    204548          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "4589    110032          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "3253     74999          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "15760   409695          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "19970   460823          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "119        147         13.0  3.743106e+39  2.879312e+38  3.402823e+38   \n",
      "20286   468377         12.0  3.743106e+39  3.119255e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "22161  3.402823e+38     255.000000          9.0  3.062541e+39  3.402823e+38   \n",
      "8473            NaN            NaN          NaN           NaN           NaN   \n",
      "12017  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "8922   3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "4589   3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "3253   3.402823e+38     255.000000          6.0  2.041694e+39  3.402823e+38   \n",
      "15760  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "19970  2.655612e+00     155.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "119    2.810641e+00     224.230769         13.0  3.743106e+39  2.879312e+38   \n",
      "20286  2.473474e+00     238.250000         12.0  3.743106e+39  3.119255e+38   \n",
      "\n",
      "       ...  NTLsumD_2011  NTLavgD_2011  NTLmaxD_2011  NTLminD_2011  \\\n",
      "22161  ...           NaN           NaN           NaN           NaN   \n",
      "8473   ...           NaN           NaN           NaN           NaN   \n",
      "12017  ...           NaN           NaN           NaN           NaN   \n",
      "8922   ...           NaN           NaN           NaN           NaN   \n",
      "4589   ...           NaN           NaN           NaN           NaN   \n",
      "3253   ...           NaN           NaN           NaN           NaN   \n",
      "15760  ...           NaN           NaN           NaN           NaN   \n",
      "19970  ...           3.0           3.0           3.0           3.0   \n",
      "119    ...           9.0           4.5           5.0           4.0   \n",
      "20286  ...           3.0           3.0           3.0           3.0   \n",
      "\n",
      "       NTLcfc_D_2011  NTLctD_2012  NTLsumD_2012  NTLavgD_2012  NTLmaxD_2012  \\\n",
      "22161            NaN          NaN           NaN           NaN           NaN   \n",
      "8473             NaN          NaN           NaN           NaN           NaN   \n",
      "12017            NaN          NaN           NaN           NaN           NaN   \n",
      "8922             NaN          NaN           NaN           NaN           NaN   \n",
      "4589             NaN          NaN           NaN           NaN           NaN   \n",
      "3253             NaN          NaN           NaN           NaN           NaN   \n",
      "15760            NaN          NaN           NaN           NaN           NaN   \n",
      "19970           75.0          1.0      4.628572      4.628572      4.628572   \n",
      "119             64.0          2.0      8.492537      4.246269      4.268657   \n",
      "20286           76.0          1.0      3.970149      3.970149      3.970149   \n",
      "\n",
      "       NTLminD_2012  \n",
      "22161           NaN  \n",
      "8473            NaN  \n",
      "12017           NaN  \n",
      "8922            NaN  \n",
      "4589            NaN  \n",
      "3253            NaN  \n",
      "15760           NaN  \n",
      "19970      4.628572  \n",
      "119        4.223881  \n",
      "20286      3.970149  \n",
      "\n",
      "[10 rows x 84 columns]\n",
      "Loading data for Msk_D_2012_cfc.tif. Sun Feb 19 10:36:04 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2012. Sun Feb 19 10:36:05 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:36:06 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:36:06 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:36:07 2023 \n",
      "\n",
      "         Z  Sett_ID\n",
      "319293  79   180351\n",
      "415036  65   247510\n",
      "437649  66   277531\n",
      "325695  69   227046\n",
      "231076  73   409797\n",
      "466111  68   161326\n",
      "357257  65   215151\n",
      "231954  72   457081\n",
      "395169  69   266152\n",
      "330099  75   180242\n",
      "\n",
      "Exported as table. Sun Feb 19 10:36:07 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:36:07 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2012. Sun Feb 19 10:36:07 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "2454     51585         20.0  6.465365e+39  3.232682e+38  3.402823e+38   \n",
      "2649     51828         30.0  9.868188e+39  3.289396e+38  3.402823e+38   \n",
      "7061    180185         14.0  4.423671e+39  3.159765e+38  3.402823e+38   \n",
      "5847    138120         63.0  2.109751e+40  3.348810e+38  3.402823e+38   \n",
      "4635    117186         26.0  7.486212e+39  2.879312e+38  3.402823e+38   \n",
      "18219   454198          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "12040   277834          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "18146   454118          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "8750    204103          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "403        474         17.0  5.784800e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "2454   2.975000e+00     245.000000         20.0  6.465365e+39  3.232682e+38   \n",
      "2649   2.762649e+00     248.300000         30.0  9.868188e+39  3.289396e+38   \n",
      "7061   2.983405e+00     240.571429         14.0  4.423671e+39  3.159765e+38   \n",
      "5847   2.543812e+00     251.777778         63.0  2.109751e+40  3.348810e+38   \n",
      "4635   3.203629e+00     223.846154         26.0  7.486212e+39  2.879312e+38   \n",
      "18219  2.607143e+00     205.250000          4.0  1.020847e+39  2.552118e+38   \n",
      "12040  2.481331e+00     153.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "18146  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "8750   3.095916e+00     221.833333          6.0  1.701412e+39  2.835686e+38   \n",
      "403    3.402823e+38     255.000000         17.0  5.784800e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2011  NTLmaxD_2011  NTLminD_2011  NTLcfc_D_2011  \\\n",
      "2454   ...          4.00           4.0           4.0          70.00   \n",
      "2649   ...          3.00           3.0           3.0          62.00   \n",
      "7061   ...          4.00           4.0           4.0          69.00   \n",
      "5847   ...          4.00           4.0           4.0          76.00   \n",
      "4635   ...          3.25           4.0           3.0          66.75   \n",
      "18219  ...          3.00           3.0           3.0          76.00   \n",
      "12040  ...          4.00           4.0           4.0          69.00   \n",
      "18146  ...           NaN           NaN           NaN            NaN   \n",
      "8750   ...          3.00           3.0           3.0          68.00   \n",
      "403    ...           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTLctD_2012  NTLsumD_2012  NTLavgD_2012  NTLmaxD_2012  NTLminD_2012  \\\n",
      "2454           1.0      4.422535      4.422535      4.422535      4.422535   \n",
      "2649           1.0      3.714286      3.714286      3.714286      3.714286   \n",
      "7061           1.0      3.768116      3.768116      3.768116      3.768116   \n",
      "5847           1.0      3.351351      3.351351      3.351351      3.351351   \n",
      "4635           4.0     13.812786      3.453196      3.597222      3.361111   \n",
      "18219          1.0      3.267606      3.267606      3.267606      3.267606   \n",
      "12040          1.0      3.695652      3.695652      3.695652      3.695652   \n",
      "18146          NaN           NaN           NaN           NaN           NaN   \n",
      "8750           1.0      3.875000      3.875000      3.875000      3.875000   \n",
      "403            NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_D_2012  \n",
      "2454           72.00  \n",
      "2649           70.00  \n",
      "7061           73.00  \n",
      "5847           76.00  \n",
      "4635           72.25  \n",
      "18219          71.00  \n",
      "12040          69.00  \n",
      "18146            NaN  \n",
      "8750           73.00  \n",
      "403              NaN  \n",
      "\n",
      "[10 rows x 85 columns]\n",
      "Loading data for Msk_D_2013_avg.tif. Sun Feb 19 10:36:09 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2013. Sun Feb 19 10:36:10 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:36:11 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:36:11 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:36:12 2023 \n",
      "\n",
      "                Z  Sett_ID\n",
      "200397   3.727273   454292\n",
      "400078   3.542857    37119\n",
      "237411   3.680556   409642\n",
      "296015   4.260870   227641\n",
      "516322   3.878788    35188\n",
      "452445  58.532467    51196\n",
      "561553   4.338461    87580\n",
      "378758   3.703125   180106\n",
      "277828   4.071429   180519\n",
      "442539   4.128205    51657\n",
      "\n",
      "Exported as table. Sun Feb 19 10:36:12 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:36:12 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2013. Sun Feb 19 10:36:12 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "19777   458473          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "14213   388878          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "17558   445757         13.0  4.083388e+39  3.141068e+38  3.402823e+38   \n",
      "518        662          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "566        727         30.0  9.868188e+39  3.289396e+38  3.402823e+38   \n",
      "13225   315033          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "4834    117420          8.0  2.041694e+39  2.552118e+38  3.402823e+38   \n",
      "19886   458942          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "969      22684         31.0  1.054875e+40  3.402823e+38  3.402823e+38   \n",
      "6883    167044         23.0  7.486212e+39  3.254875e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "19777  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "14213  3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "17558  2.526251e+00     239.846154         13.0  4.083388e+39  3.141068e+38   \n",
      "518    3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "566    2.795977e+00     248.033333         30.0  9.868188e+39  3.289396e+38   \n",
      "13225  2.631263e+00     221.666667          6.0  1.701412e+39  2.835686e+38   \n",
      "4834   2.636329e+00     204.250000          8.0  2.041694e+39  2.552118e+38   \n",
      "19886  2.559783e+00     226.428571          7.0  2.041694e+39  2.916706e+38   \n",
      "969    3.402823e+38     255.000000         31.0  1.054875e+40  3.402823e+38   \n",
      "6883   2.805644e+00     246.260870         23.0  7.486212e+39  3.254875e+38   \n",
      "\n",
      "       ...  NTLsumD_2012  NTLavgD_2012  NTLmaxD_2012  NTLminD_2012  \\\n",
      "19777  ...           NaN           NaN           NaN           NaN   \n",
      "14213  ...           NaN           NaN           NaN           NaN   \n",
      "17558  ...      3.405406      3.405406      3.405406      3.405406   \n",
      "518    ...           NaN           NaN           NaN           NaN   \n",
      "566    ...      4.104477      4.104477      4.104477      4.104477   \n",
      "13225  ...      3.838235      3.838235      3.838235      3.838235   \n",
      "4834   ...      6.820083      3.410041      3.414286      3.405797   \n",
      "19886  ...      3.358209      3.358209      3.358209      3.358209   \n",
      "969    ...           NaN           NaN           NaN           NaN   \n",
      "6883   ...      3.606061      3.606061      3.606061      3.606061   \n",
      "\n",
      "       NTLcfc_D_2012  NTLctD_2013  NTLsumD_2013  NTLavgD_2013  NTLmaxD_2013  \\\n",
      "19777            NaN          NaN           NaN           NaN           NaN   \n",
      "14213            NaN          NaN           NaN           NaN           NaN   \n",
      "17558           74.0          1.0      3.380282      3.380282      3.380282   \n",
      "518              NaN          NaN           NaN           NaN           NaN   \n",
      "566             72.0          1.0      4.571429      4.571429      4.571429   \n",
      "13225           69.0          1.0      3.888889      3.888889      3.888889   \n",
      "4834            73.0          2.0      7.121469      3.560735      3.569231   \n",
      "19886           67.0          1.0      3.516129      3.516129      3.516129   \n",
      "969              NaN          NaN           NaN           NaN           NaN   \n",
      "6883            66.0          1.0      4.000000      4.000000      4.000000   \n",
      "\n",
      "       NTLminD_2013  \n",
      "19777           NaN  \n",
      "14213           NaN  \n",
      "17558      3.380282  \n",
      "518             NaN  \n",
      "566        4.571429  \n",
      "13225      3.888889  \n",
      "4834       3.552239  \n",
      "19886      3.516129  \n",
      "969             NaN  \n",
      "6883       4.000000  \n",
      "\n",
      "[10 rows x 90 columns]\n",
      "Loading data for Msk_D_2013_cfc.tif. Sun Feb 19 10:36:14 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2013. Sun Feb 19 10:36:15 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:36:16 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:36:16 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:36:17 2023 \n",
      "\n",
      "         Z  Sett_ID\n",
      "434753  65   236712\n",
      "349033  66   180312\n",
      "479349  68    30231\n",
      "256160  63   396304\n",
      "349728  80   117342\n",
      "288535  77   388586\n",
      "485821  73    51508\n",
      "315767  69   212707\n",
      "225764  76   415487\n",
      "434083  64   277953\n",
      "\n",
      "Exported as table. Sun Feb 19 10:36:17 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:36:17 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2013. Sun Feb 19 10:36:17 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "16257   415996         55.0  1.837525e+40  3.340954e+38  3.402823e+38   \n",
      "11671   271206          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "341        396         18.0  5.104235e+39  2.835686e+38  3.402823e+38   \n",
      "4890    117508        104.0  3.504908e+40  3.370104e+38  3.402823e+38   \n",
      "2459     51590         13.0  4.083388e+39  3.141068e+38  3.402823e+38   \n",
      "11753   276468         61.0  2.041694e+40  3.347039e+38  3.402823e+38   \n",
      "20823   474701          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "11647   270377         31.0  1.054875e+40  3.402823e+38  3.402823e+38   \n",
      "4160     98184         14.0  4.423671e+39  3.159765e+38  3.402823e+38   \n",
      "7686    186834          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "16257  2.556373e+00     251.454545         55.0  1.837525e+40  3.340954e+38   \n",
      "11671  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "341    2.828218e+00     220.833333         18.0  5.104235e+39  2.835686e+38   \n",
      "4890   2.485169e+00     253.028846        104.0  3.504908e+40  3.370104e+38   \n",
      "2459   2.970481e+00     239.307692         13.0  4.083388e+39  3.141068e+38   \n",
      "11753  2.744880e+00     251.770492         61.0  2.041694e+40  3.347039e+38   \n",
      "20823  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "11647  3.402823e+38     255.000000         31.0  1.054875e+40  3.402823e+38   \n",
      "4160   2.610090e+00     240.571429         14.0  4.423671e+39  3.159765e+38   \n",
      "7686   3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2012  NTLmaxD_2012  NTLminD_2012  NTLcfc_D_2012  \\\n",
      "16257  ...      3.750000      3.750000      3.750000           77.0   \n",
      "11671  ...           NaN           NaN           NaN            NaN   \n",
      "341    ...      3.782407      3.875000      3.722222           73.0   \n",
      "4890   ...      3.750000      3.750000      3.750000           74.0   \n",
      "2459   ...      3.774648      3.774648      3.774648           72.0   \n",
      "11753  ...      3.358209      3.358209      3.358209           67.0   \n",
      "20823  ...           NaN           NaN           NaN            NaN   \n",
      "11647  ...           NaN           NaN           NaN            NaN   \n",
      "4160   ...      3.761905      3.761905      3.761905           63.0   \n",
      "7686   ...           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTLctD_2013  NTLsumD_2013  NTLavgD_2013  NTLmaxD_2013  NTLminD_2013  \\\n",
      "16257          1.0      3.371428      3.371428      3.371428      3.371428   \n",
      "11671          NaN           NaN           NaN           NaN           NaN   \n",
      "341            3.0     11.816839      3.938946      4.013699      3.861111   \n",
      "4890           1.0      4.000000      4.000000      4.000000      4.000000   \n",
      "2459           1.0      3.680000      3.680000      3.680000      3.680000   \n",
      "11753          1.0      3.648649      3.648649      3.648649      3.648649   \n",
      "20823          NaN           NaN           NaN           NaN           NaN   \n",
      "11647          NaN           NaN           NaN           NaN           NaN   \n",
      "4160           1.0      3.687500      3.687500      3.687500      3.687500   \n",
      "7686           NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_D_2013  \n",
      "16257      70.000000  \n",
      "11671            NaN  \n",
      "341        72.333333  \n",
      "4890       73.000000  \n",
      "2459       79.000000  \n",
      "11753      75.000000  \n",
      "20823            NaN  \n",
      "11647            NaN  \n",
      "4160       67.000000  \n",
      "7686             NaN  \n",
      "\n",
      "[10 rows x 91 columns]\n",
      "Loading data for Msk_V_2012_avg.tif. Sun Feb 19 10:36:19 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2012. Sun Feb 19 10:36:23 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:36:26 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:36:26 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:36:28 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "1274228   3.616046   227047\n",
      "1560696   0.201957   265667\n",
      "867092    0.127327   456898\n",
      "1283495   0.187434   309529\n",
      "1810573   0.163434   155897\n",
      "1209382   0.426780   227047\n",
      "867096    0.058891   456898\n",
      "1806685   8.331445    51196\n",
      "922673    2.750182   400483\n",
      "1231017  14.372094   227047\n",
      "\n",
      "Exported as table. Sun Feb 19 10:36:28 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:36:28 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2012. Sun Feb 19 10:36:28 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "8440    203588          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "22850   574874          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "21220   492964          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "22902   580960          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "21723   508785          9.0  3.062541e+39  3.402823e+38  3.402823e+38   \n",
      "7113    180246          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "16585   416665          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "18163   454135          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "14655   399800          8.0  2.381976e+39  2.977471e+38  3.402823e+38   \n",
      "9445    215347          NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "8440   2.678121e+00        206.000          4.0  1.020847e+39  2.552118e+38   \n",
      "22850  3.402823e+38        255.000          2.0  6.805647e+38  3.402823e+38   \n",
      "21220  2.843378e+00        204.500          4.0  1.020847e+39  2.552118e+38   \n",
      "22902  2.493350e+00        230.250          8.0  2.381976e+39  2.977471e+38   \n",
      "21723  3.402823e+38        255.000          9.0  3.062541e+39  3.402823e+38   \n",
      "7113   3.090022e+00        226.000          7.0  2.041694e+39  2.916706e+38   \n",
      "16585  3.402823e+38        255.000          5.0  1.701412e+39  3.402823e+38   \n",
      "18163  2.753508e+00        189.000          3.0  6.805647e+38  2.268549e+38   \n",
      "14655  2.636755e+00        229.625          8.0  2.381976e+39  2.977471e+38   \n",
      "9445            NaN            NaN          NaN           NaN           NaN   \n",
      "\n",
      "       ...  NTLsumD_2013  NTLavgD_2013  NTLmaxD_2013  NTLminD_2013  \\\n",
      "8440   ...      3.508197      3.508197      3.508197      3.508197   \n",
      "22850  ...           NaN           NaN           NaN           NaN   \n",
      "21220  ...      3.434783      3.434783      3.434783      3.434783   \n",
      "22902  ...      3.227273      3.227273      3.227273      3.227273   \n",
      "21723  ...           NaN           NaN           NaN           NaN   \n",
      "7113   ...      3.845070      3.845070      3.845070      3.845070   \n",
      "16585  ...           NaN           NaN           NaN           NaN   \n",
      "18163  ...      3.507692      3.507692      3.507692      3.507692   \n",
      "14655  ...      3.397260      3.397260      3.397260      3.397260   \n",
      "9445   ...           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_D_2013  NTLctV_2012  NTLsumV_2012  NTLavgV_2012  NTLmaxV_2012  \\\n",
      "8440            62.0          NaN           NaN           NaN           NaN   \n",
      "22850            NaN          NaN           NaN           NaN           NaN   \n",
      "21220           69.0          NaN           NaN           NaN           NaN   \n",
      "22902           66.0          NaN           NaN           NaN           NaN   \n",
      "21723            NaN          NaN           NaN           NaN           NaN   \n",
      "7113            74.0          NaN           NaN           NaN           NaN   \n",
      "16585            NaN          NaN           NaN           NaN           NaN   \n",
      "18163           65.0          NaN           NaN           NaN           NaN   \n",
      "14655           73.0          NaN           NaN           NaN           NaN   \n",
      "9445             NaN          NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLminV_2012  \n",
      "8440            NaN  \n",
      "22850           NaN  \n",
      "21220           NaN  \n",
      "22902           NaN  \n",
      "21723           NaN  \n",
      "7113            NaN  \n",
      "16585           NaN  \n",
      "18163           NaN  \n",
      "14655           NaN  \n",
      "9445            NaN  \n",
      "\n",
      "[10 rows x 96 columns]\n",
      "Loading data for Msk_V_2012_cfc.tif. Sun Feb 19 10:36:30 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2012. Sun Feb 19 10:36:33 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:36:37 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:36:37 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:36:41 2023 \n",
      "\n",
      "          Z  Sett_ID\n",
      "2056919  44        8\n",
      "1597911  42   102540\n",
      "463961   54   475024\n",
      "654622   60   445605\n",
      "1530072  54   265974\n",
      "950117   45   416060\n",
      "585997   51   458248\n",
      "1021790  47   396587\n",
      "1111886  45   180521\n",
      "1238005  46   203951\n",
      "\n",
      "Exported as table. Sun Feb 19 10:36:41 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:36:41 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2012. Sun Feb 19 10:36:41 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "12295   294473          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "17176   445221          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "1779     37078         14.0  4.763953e+39  3.402823e+38  3.402823e+38   \n",
      "21225   493540          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "6190    155995          5.0  1.361129e+39  2.722259e+38  3.402823e+38   \n",
      "9649    215815          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "20365   468493         10.0  3.402823e+39  3.402823e+38  3.402823e+38   \n",
      "17677   445972          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "7270    180439          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "4936    117577         13.0  4.423671e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "12295  2.772398e+00     221.833333          6.0  1.701412e+39  2.835686e+38   \n",
      "17176  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "1779   3.402823e+38     255.000000         14.0  4.763953e+39  3.402823e+38   \n",
      "21225  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "6190   2.642276e+00     214.400000          5.0  1.361129e+39  2.722259e+38   \n",
      "9649   2.966045e+00     190.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "20365  3.402823e+38     255.000000         10.0  3.402823e+39  3.402823e+38   \n",
      "17677  2.635440e+00     226.714286          7.0  2.041694e+39  2.916706e+38   \n",
      "7270   2.738461e+00     206.000000          4.0  1.020847e+39  2.552118e+38   \n",
      "4936   3.402823e+38     255.000000         13.0  4.423671e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgD_2013  NTLmaxD_2013  NTLminD_2013  NTLcfc_D_2013  \\\n",
      "12295  ...      3.757143      3.757143      3.757143           71.0   \n",
      "17176  ...           NaN           NaN           NaN            NaN   \n",
      "1779   ...           NaN           NaN           NaN            NaN   \n",
      "21225  ...           NaN           NaN           NaN            NaN   \n",
      "6190   ...      3.767123      3.767123      3.767123           75.0   \n",
      "9649   ...      4.424657      4.424657      4.424657           73.0   \n",
      "20365  ...           NaN           NaN           NaN            NaN   \n",
      "17677  ...      3.263889      3.263889      3.263889           72.0   \n",
      "7270   ...      4.046154      4.046154      4.046154           68.0   \n",
      "4936   ...           NaN           NaN           NaN            NaN   \n",
      "\n",
      "       NTLctV_2012  NTLsumV_2012  NTLavgV_2012  NTLmaxV_2012  NTLminV_2012  \\\n",
      "12295          NaN           NaN           NaN           NaN           NaN   \n",
      "17176          NaN           NaN           NaN           NaN           NaN   \n",
      "1779           NaN           NaN           NaN           NaN           NaN   \n",
      "21225          NaN           NaN           NaN           NaN           NaN   \n",
      "6190           NaN           NaN           NaN           NaN           NaN   \n",
      "9649           NaN           NaN           NaN           NaN           NaN   \n",
      "20365          NaN           NaN           NaN           NaN           NaN   \n",
      "17677          NaN           NaN           NaN           NaN           NaN   \n",
      "7270           NaN           NaN           NaN           NaN           NaN   \n",
      "4936           NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2012  \n",
      "12295           53.0  \n",
      "17176           49.0  \n",
      "1779            38.0  \n",
      "21225            NaN  \n",
      "6190            44.0  \n",
      "9649            60.5  \n",
      "20365           57.0  \n",
      "17677           56.0  \n",
      "7270            48.0  \n",
      "4936            45.0  \n",
      "\n",
      "[10 rows x 97 columns]\n",
      "Loading data for Msk_V_2013_avg.tif. Sun Feb 19 10:36:43 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2013. Sun Feb 19 10:36:47 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:36:50 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:36:50 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:36:52 2023 \n",
      "\n",
      "                Z  Sett_ID\n",
      "1270406  0.420260   180085\n",
      "1241831  1.044975   227047\n",
      "1240035  1.889953   227047\n",
      "951404   0.069640   401297\n",
      "1227401  6.547158   227047\n",
      "1290435  1.172861   227047\n",
      "1322624  0.492216   180694\n",
      "1382734  0.224199   318632\n",
      "1239991  6.156235   227047\n",
      "1199942  0.254382   137698\n",
      "\n",
      "Exported as table. Sun Feb 19 10:36:52 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:36:52 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2013. Sun Feb 19 10:36:52 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "1966     38637          6.0  1.701412e+39  2.835686e+38  3.402823e+38   \n",
      "12979   310078          5.0  1.701412e+39  3.402823e+38  3.402823e+38   \n",
      "1366     30213          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "18534   456939          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "20677   474480         12.0  4.083388e+39  3.402823e+38  3.402823e+38   \n",
      "21255   495479         19.0  6.125082e+39  3.223727e+38  3.402823e+38   \n",
      "19024   457561          1.0  2.422059e+00  2.422059e+00  2.422059e+00   \n",
      "7665    186512          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "1944     37274       1294.0  4.399851e+41  3.400194e+38  3.402823e+38   \n",
      "10603   236699         31.0  9.868188e+39  3.183286e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "1966   3.155841e+00     221.166667          6.0  1.701412e+39  2.835686e+38   \n",
      "12979  3.402823e+38     255.000000          5.0  1.701412e+39  3.402823e+38   \n",
      "1366   3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "18534  3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "20677  3.402823e+38     255.000000         12.0  4.083388e+39  3.402823e+38   \n",
      "21255  2.551923e+00     244.421053         19.0  6.125082e+39  3.223727e+38   \n",
      "19024  2.422059e+00      59.000000          1.0  2.711394e+00  2.711394e+00   \n",
      "7665   3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "1944   2.724664e+00     254.844668       1294.0  4.399851e+41  3.400194e+38   \n",
      "10603  2.673790e+00     242.161290         31.0  9.868188e+39  3.183286e+38   \n",
      "\n",
      "       ...  NTLsumV_2012  NTLavgV_2012  NTLmaxV_2012  NTLminV_2012  \\\n",
      "1966   ...      0.314439      0.157219      0.261149       0.05329   \n",
      "12979  ...           NaN           NaN           NaN           NaN   \n",
      "1366   ...           NaN           NaN           NaN           NaN   \n",
      "18534  ...           NaN           NaN           NaN           NaN   \n",
      "20677  ...           NaN           NaN           NaN           NaN   \n",
      "21255  ...           NaN           NaN           NaN           NaN   \n",
      "19024  ...           NaN           NaN           NaN           NaN   \n",
      "7665   ...           NaN           NaN           NaN           NaN   \n",
      "1944   ...           NaN           NaN           NaN           NaN   \n",
      "10603  ...           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2012  NTLctV_2013  NTLsumV_2013  NTLavgV_2013  NTLmaxV_2013  \\\n",
      "1966       52.000000          2.0      0.908269      0.454134      0.667223   \n",
      "12979      58.500000          NaN           NaN           NaN           NaN   \n",
      "1366       39.000000          NaN           NaN           NaN           NaN   \n",
      "18534      55.500000          NaN           NaN           NaN           NaN   \n",
      "20677      57.000000          NaN           NaN           NaN           NaN   \n",
      "21255      59.000000          NaN           NaN           NaN           NaN   \n",
      "19024      56.000000          NaN           NaN           NaN           NaN   \n",
      "7665       45.000000          NaN           NaN           NaN           NaN   \n",
      "1944       42.800000          NaN           NaN           NaN           NaN   \n",
      "10603      50.833333          NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLminV_2013  \n",
      "1966       0.241045  \n",
      "12979           NaN  \n",
      "1366            NaN  \n",
      "18534           NaN  \n",
      "20677           NaN  \n",
      "21255           NaN  \n",
      "19024           NaN  \n",
      "7665            NaN  \n",
      "1944            NaN  \n",
      "10603           NaN  \n",
      "\n",
      "[10 rows x 102 columns]\n",
      "Loading data for Msk_V_2013_cfc.tif. Sun Feb 19 10:36:54 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2013. Sun Feb 19 10:36:57 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:37:01 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:37:01 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:37:05 2023 \n",
      "\n",
      "           Z  Sett_ID\n",
      "692422   107   458062\n",
      "1400489  111   294245\n",
      "1639714  105   188470\n",
      "631037   104   458183\n",
      "2046110   95        7\n",
      "1301254  111   227047\n",
      "997277   103   528163\n",
      "1659050  108    51187\n",
      "1611149  106   266112\n",
      "1302916  112   203483\n",
      "\n",
      "Exported as table. Sun Feb 19 10:37:05 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:37:05 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2013. Sun Feb 19 10:37:05 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "17619   445870          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "16022   414290          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "7100    180233         51.0  1.701412e+40  3.336101e+38  3.402823e+38   \n",
      "15198   408820          6.0  1.361129e+39  2.268549e+38  3.402823e+38   \n",
      "18051   454008          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "23343   597223        112.0  3.811162e+40  3.402823e+38  3.402823e+38   \n",
      "2496     51646          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "8803    204266          4.0  1.361129e+39  3.402823e+38  3.402823e+38   \n",
      "21389   501664          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "1606     36872         18.0  6.125082e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "17619  2.683971e+00     226.714286          7.0  2.041694e+39  2.916706e+38   \n",
      "16022  2.528690e+00     157.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "7100   2.746951e+00     251.039216         51.0  1.701412e+40  3.336101e+38   \n",
      "15198  2.557358e+00     188.500000          6.0  1.361129e+39  2.268549e+38   \n",
      "18051  2.550189e+00     189.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "23343  3.402823e+38     255.000000        112.0  3.811162e+40  3.402823e+38   \n",
      "2496   3.402823e+38     255.000000          1.0  3.402823e+38  3.402823e+38   \n",
      "8803   3.402823e+38     255.000000          4.0  1.361129e+39  3.402823e+38   \n",
      "21389  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "1606   3.402823e+38     255.000000         18.0  6.125082e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgV_2012  NTLmaxV_2012  NTLminV_2012  NTLcfc_V_2012  \\\n",
      "17619  ...           NaN           NaN           NaN      53.500000   \n",
      "16022  ...           NaN           NaN           NaN      53.000000   \n",
      "7100   ...           NaN           NaN           NaN      41.714286   \n",
      "15198  ...           NaN           NaN           NaN      48.153846   \n",
      "18051  ...           NaN           NaN           NaN      52.000000   \n",
      "23343  ...           NaN           NaN           NaN      51.000000   \n",
      "2496   ...           NaN           NaN           NaN      46.000000   \n",
      "8803   ...           NaN           NaN           NaN      41.666667   \n",
      "21389  ...           NaN           NaN           NaN      51.666667   \n",
      "1606   ...           NaN           NaN           NaN      49.000000   \n",
      "\n",
      "       NTLctV_2013  NTLsumV_2013  NTLavgV_2013  NTLmaxV_2013  NTLminV_2013  \\\n",
      "17619          NaN           NaN           NaN           NaN           NaN   \n",
      "16022          NaN           NaN           NaN           NaN           NaN   \n",
      "7100           NaN           NaN           NaN           NaN           NaN   \n",
      "15198          NaN           NaN           NaN           NaN           NaN   \n",
      "18051          NaN           NaN           NaN           NaN           NaN   \n",
      "23343          NaN           NaN           NaN           NaN           NaN   \n",
      "2496           NaN           NaN           NaN           NaN           NaN   \n",
      "8803           NaN           NaN           NaN           NaN           NaN   \n",
      "21389          NaN           NaN           NaN           NaN           NaN   \n",
      "1606           NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2013  \n",
      "17619     113.500000  \n",
      "16022     123.000000  \n",
      "7100      104.285714  \n",
      "15198     118.230769  \n",
      "18051     114.000000  \n",
      "23343     119.000000  \n",
      "2496      106.000000  \n",
      "8803      116.666667  \n",
      "21389     117.000000  \n",
      "1606      104.000000  \n",
      "\n",
      "[10 rows x 103 columns]\n",
      "Loading data for Msk_V_2014_avg.tif. Sun Feb 19 10:37:07 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2014. Sun Feb 19 10:37:11 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:37:15 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:37:15 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:37:16 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "685094    4.053637   456905\n",
      "1212988   3.282514   227047\n",
      "692296    3.905006   456905\n",
      "1274230   7.839927   234081\n",
      "1829465   0.462932   351201\n",
      "1812373   0.681269   155897\n",
      "694099    1.974902   456905\n",
      "1292191   0.250984   227277\n",
      "1562495   2.550558   265667\n",
      "1240019  21.193678   227047\n",
      "\n",
      "Exported as table. Sun Feb 19 10:37:16 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:37:16 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2014. Sun Feb 19 10:37:16 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "23090   581358         12.0  4.083388e+39  3.402823e+38  3.402823e+38   \n",
      "20867   474787         45.0  1.531271e+40  3.402823e+38  3.402823e+38   \n",
      "3575     87124         44.0  1.497242e+40  3.402823e+38  3.402823e+38   \n",
      "6790    166023         10.0  3.062541e+39  3.062541e+38  3.402823e+38   \n",
      "3445     84315         54.0  1.803496e+40  3.339808e+38  3.402823e+38   \n",
      "6261    156113          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "1875     37189         25.0  8.166776e+39  3.266711e+38  3.402823e+38   \n",
      "20951   474958          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "9565    215630          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "14646   399221         10.0  3.402823e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "23090  3.402823e+38     255.000000         12.0  4.083388e+39  3.402823e+38   \n",
      "20867  3.402823e+38     255.000000         45.0  1.531271e+40  3.402823e+38   \n",
      "3575   3.402823e+38     255.000000         44.0  1.497242e+40  3.402823e+38   \n",
      "6790   2.959375e+00     235.000000         10.0  3.062541e+39  3.062541e+38   \n",
      "3445   2.709792e+00     251.203704         54.0  1.803496e+40  3.339808e+38   \n",
      "6261   3.402823e+38     255.000000          6.0  2.041694e+39  3.402823e+38   \n",
      "1875   2.573276e+00     246.800000         25.0  8.166776e+39  3.266711e+38   \n",
      "20951  2.912880e+00     188.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "9565   2.845031e+00     206.000000          4.0  1.020847e+39  2.552118e+38   \n",
      "14646  3.402823e+38     255.000000         10.0  3.402823e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumV_2013  NTLavgV_2013  NTLmaxV_2013  NTLminV_2013  \\\n",
      "23090  ...           NaN           NaN           NaN           NaN   \n",
      "20867  ...           NaN           NaN           NaN           NaN   \n",
      "3575   ...           NaN           NaN           NaN           NaN   \n",
      "6790   ...           NaN           NaN           NaN           NaN   \n",
      "3445   ...           NaN           NaN           NaN           NaN   \n",
      "6261   ...           NaN           NaN           NaN           NaN   \n",
      "1875   ...           NaN           NaN           NaN           NaN   \n",
      "20951  ...           NaN           NaN           NaN           NaN   \n",
      "9565   ...           NaN           NaN           NaN           NaN   \n",
      "14646  ...           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2013  NTLctV_2014  NTLsumV_2014  NTLavgV_2014  NTLmaxV_2014  \\\n",
      "23090     130.500000          NaN           NaN           NaN           NaN   \n",
      "20867     116.000000          NaN           NaN           NaN           NaN   \n",
      "3575       97.666667          NaN           NaN           NaN           NaN   \n",
      "6790      101.000000          NaN           NaN           NaN           NaN   \n",
      "3445       92.000000          NaN           NaN           NaN           NaN   \n",
      "6261       96.666667          NaN           NaN           NaN           NaN   \n",
      "1875      104.750000          NaN           NaN           NaN           NaN   \n",
      "20951     116.000000          NaN           NaN           NaN           NaN   \n",
      "9565      108.500000          NaN           NaN           NaN           NaN   \n",
      "14646     101.000000          NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLminV_2014  \n",
      "23090           NaN  \n",
      "20867           NaN  \n",
      "3575            NaN  \n",
      "6790            NaN  \n",
      "3445            NaN  \n",
      "6261            NaN  \n",
      "1875            NaN  \n",
      "20951           NaN  \n",
      "9565            NaN  \n",
      "14646           NaN  \n",
      "\n",
      "[10 rows x 108 columns]\n",
      "Loading data for Msk_V_2014_cfc.tif. Sun Feb 19 10:37:18 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2014. Sun Feb 19 10:37:21 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:37:25 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:37:25 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:37:29 2023 \n",
      "\n",
      "           Z  Sett_ID\n",
      "1272454  101   227043\n",
      "1907678   96    51540\n",
      "1308684   99   294532\n",
      "1229803  108   319180\n",
      "1895177   98    98104\n",
      "1799476  113    51196\n",
      "1439944  101   215558\n",
      "1267006  106   227047\n",
      "465365   114   468544\n",
      "413100   114   469627\n",
      "\n",
      "Exported as table. Sun Feb 19 10:37:29 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:37:29 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2014. Sun Feb 19 10:37:29 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "18368   454361         11.0  3.402823e+39  3.093476e+38  3.402823e+38   \n",
      "18411   454623          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "7707    187039          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "9017    205885          NaN           NaN           NaN           NaN   \n",
      "15459   409249          5.0  1.361129e+39  2.722259e+38  3.402823e+38   \n",
      "9535    215560          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "381        446         25.0  8.166776e+39  3.266711e+38  3.402823e+38   \n",
      "19578   458234          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "21198   487769          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "10774   247225         11.0  3.743106e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "18368  2.543651e+00     236.909091         11.0  3.402823e+39  3.093476e+38   \n",
      "18411  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "7707   2.694623e+00     226.285714          7.0  2.041694e+39  2.916706e+38   \n",
      "9017            NaN            NaN          NaN           NaN           NaN   \n",
      "15459  2.540616e+00     215.600000          5.0  1.361129e+39  2.722259e+38   \n",
      "9535   2.788301e+00     189.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "381    2.578030e+00     246.800000         25.0  8.166776e+39  3.266711e+38   \n",
      "19578  2.528602e+00     154.500000          2.0  3.402823e+38  1.701412e+38   \n",
      "21198  3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "10774  3.402823e+38     255.000000         11.0  3.743106e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLavgV_2013  NTLmaxV_2013  NTLminV_2013  NTLcfc_V_2013  \\\n",
      "18368  ...           NaN           NaN           NaN         113.00   \n",
      "18411  ...           NaN           NaN           NaN         109.00   \n",
      "7707   ...           NaN           NaN           NaN         107.00   \n",
      "9017   ...           NaN           NaN           NaN         103.00   \n",
      "15459  ...           NaN           NaN           NaN         106.75   \n",
      "9535   ...           NaN           NaN           NaN         109.50   \n",
      "381    ...           NaN           NaN           NaN          96.00   \n",
      "19578  ...           NaN           NaN           NaN         104.00   \n",
      "21198  ...           NaN           NaN           NaN         114.00   \n",
      "10774  ...           NaN           NaN           NaN          98.50   \n",
      "\n",
      "       NTLctV_2014  NTLsumV_2014  NTLavgV_2014  NTLmaxV_2014  NTLminV_2014  \\\n",
      "18368          NaN           NaN           NaN           NaN           NaN   \n",
      "18411          NaN           NaN           NaN           NaN           NaN   \n",
      "7707           NaN           NaN           NaN           NaN           NaN   \n",
      "9017           NaN           NaN           NaN           NaN           NaN   \n",
      "15459          NaN           NaN           NaN           NaN           NaN   \n",
      "9535           NaN           NaN           NaN           NaN           NaN   \n",
      "381            NaN           NaN           NaN           NaN           NaN   \n",
      "19578          NaN           NaN           NaN           NaN           NaN   \n",
      "21198          NaN           NaN           NaN           NaN           NaN   \n",
      "10774          NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2014  \n",
      "18368         117.00  \n",
      "18411         112.00  \n",
      "7707          110.00  \n",
      "9017          109.00  \n",
      "15459         116.75  \n",
      "9535           97.50  \n",
      "381           101.40  \n",
      "19578         121.00  \n",
      "21198         119.00  \n",
      "10774          94.50  \n",
      "\n",
      "[10 rows x 109 columns]\n",
      "Loading data for Msk_V_2015_avg.tif. Sun Feb 19 10:37:32 2023 \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished gdal.Translate() for year 2015. Sun Feb 19 10:37:35 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:37:39 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:37:39 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:37:40 2023 \n",
      "\n",
      "                 Z  Sett_ID\n",
      "1385940   0.250107   215150\n",
      "1813090   0.869642   277234\n",
      "1794048   0.959889    51640\n",
      "1144605   0.587136   501405\n",
      "1274244   2.391495   227047\n",
      "1848094   0.383477    51464\n",
      "1265228  21.848469   227047\n",
      "1250831   2.345626   227047\n",
      "2231581   0.377271        3\n",
      "1819295   5.830219    51196\n",
      "\n",
      "Exported as table. Sun Feb 19 10:37:40 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:37:40 2023 \n",
      "\n",
      "\n",
      "Desired aggregation methods applied to settlement level, year 2015. Sun Feb 19 10:37:40 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "8455    203615          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "18347   454337          7.0  2.041694e+39  2.916706e+38  3.402823e+38   \n",
      "7795    188429         26.0  8.507059e+39  3.271946e+38  3.402823e+38   \n",
      "5286    129754         40.0  1.361129e+40  3.402823e+38  3.402823e+38   \n",
      "6511    162562          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "20664   474462         22.0  7.486212e+39  3.402823e+38  3.402823e+38   \n",
      "700      10824         16.0  5.104235e+39  3.190147e+38  3.402823e+38   \n",
      "14733   400575         55.0  1.837525e+40  3.340954e+38  3.402823e+38   \n",
      "12777   309774          2.0  3.402823e+38  1.701412e+38  3.402823e+38   \n",
      "3945     93263          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "8455   2.789826e+00     206.250000          4.0  1.020847e+39  2.552118e+38   \n",
      "18347  2.459195e+00     226.285714          7.0  2.041694e+39  2.916706e+38   \n",
      "7795   2.889969e+00     247.461538         26.0  8.507059e+39  3.271946e+38   \n",
      "5286   3.402823e+38     255.000000         40.0  1.361129e+40  3.402823e+38   \n",
      "6511   3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "20664  3.402823e+38     255.000000         22.0  7.486212e+39  3.402823e+38   \n",
      "700    2.710331e+00     242.312500         16.0  5.104235e+39  3.190147e+38   \n",
      "14733  2.818326e+00     251.345455         55.0  1.837525e+40  3.340954e+38   \n",
      "12777  2.640787e+00     155.000000          2.0  3.402823e+38  1.701412e+38   \n",
      "3945   3.402823e+38     255.000000          3.0  1.020847e+39  3.402823e+38   \n",
      "\n",
      "       ...  NTLsumV_2014  NTLavgV_2014  NTLmaxV_2014  NTLminV_2014  \\\n",
      "8455   ...           NaN           NaN           NaN           NaN   \n",
      "18347  ...           NaN           NaN           NaN           NaN   \n",
      "7795   ...           NaN           NaN           NaN           NaN   \n",
      "5286   ...           NaN           NaN           NaN           NaN   \n",
      "6511   ...           NaN           NaN           NaN           NaN   \n",
      "20664  ...           NaN           NaN           NaN           NaN   \n",
      "700    ...           NaN           NaN           NaN           NaN   \n",
      "14733  ...           NaN           NaN           NaN           NaN   \n",
      "12777  ...           NaN           NaN           NaN           NaN   \n",
      "3945   ...           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2014  NTLctV_2015  NTLsumV_2015  NTLavgV_2015  NTLmaxV_2015  \\\n",
      "8455          109.00          NaN           NaN           NaN           NaN   \n",
      "18347         123.75          NaN           NaN           NaN           NaN   \n",
      "7795           92.00          NaN           NaN           NaN           NaN   \n",
      "5286           94.50          NaN           NaN           NaN           NaN   \n",
      "6511           98.00          NaN           NaN           NaN           NaN   \n",
      "20664         116.00          NaN           NaN           NaN           NaN   \n",
      "700            94.50          NaN           NaN           NaN           NaN   \n",
      "14733         110.00          NaN           NaN           NaN           NaN   \n",
      "12777          97.00          NaN           NaN           NaN           NaN   \n",
      "3945           94.00          NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLminV_2015  \n",
      "8455            NaN  \n",
      "18347           NaN  \n",
      "7795            NaN  \n",
      "5286            NaN  \n",
      "6511            NaN  \n",
      "20664           NaN  \n",
      "700             NaN  \n",
      "14733           NaN  \n",
      "12777           NaN  \n",
      "3945            NaN  \n",
      "\n",
      "[10 rows x 114 columns]\n",
      "Loading data for Msk_V_2015_cfc.tif. Sun Feb 19 10:37:42 2023 \n",
      "\n",
      "Finished gdal.Translate() for year 2015. Sun Feb 19 10:37:45 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Sun Feb 19 10:37:49 2023 \n",
      "\n",
      "Created geodataframe from non-NoData points. Sun Feb 19 10:37:49 2023 \n",
      "\n",
      "\n",
      "Joined zone ID onto vectorized raster cells. Sun Feb 19 10:37:53 2023 \n",
      "\n",
      "           Z  Sett_ID\n",
      "580799   118   445691\n",
      "1137161  112   180646\n",
      "542966   112   474205\n",
      "390245   121   581470\n",
      "1564294  106   265667\n",
      "643670   107   458172\n",
      "1576343   98   102546\n",
      "2456828   94       48\n",
      "1229077  107   210283\n",
      "1501381  106   247086\n",
      "\n",
      "Exported as table. Sun Feb 19 10:37:53 2023 \n",
      "\n",
      "Removed (or skipped if error) intermediate xyz file. Sun Feb 19 10:37:53 2023 \n",
      "\n",
      "\n",
      "Count of cloud-free observations averaged to settlement level, year 2015. Sun Feb 19 10:37:53 2023 \n",
      "\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "16631   416841         24.0  8.166776e+39  3.402823e+38  3.402823e+38   \n",
      "8650    203899          1.0  3.402823e+38  3.402823e+38  3.402823e+38   \n",
      "8662    203921          3.0  1.020847e+39  3.402823e+38  3.402823e+38   \n",
      "4073     96146          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "15173   405453          5.0  1.361129e+39  2.722259e+38  3.402823e+38   \n",
      "15095   404545          6.0  2.041694e+39  3.402823e+38  3.402823e+38   \n",
      "17672   445962          9.0  3.062541e+39  3.402823e+38  3.402823e+38   \n",
      "18628   457045         10.0  3.402823e+39  3.402823e+38  3.402823e+38   \n",
      "8298    203357          1.0  2.941716e+00  2.941716e+00  2.941716e+00   \n",
      "8440    203588          4.0  1.020847e+39  2.552118e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "16631  3.402823e+38          255.0         24.0  8.166776e+39  3.402823e+38   \n",
      "8650   3.402823e+38          255.0          1.0  3.402823e+38  3.402823e+38   \n",
      "8662   3.402823e+38          255.0          3.0  1.020847e+39  3.402823e+38   \n",
      "4073   3.402823e+38          255.0          8.0  2.722259e+39  3.402823e+38   \n",
      "15173  2.989130e+00          215.6          5.0  1.361129e+39  2.722259e+38   \n",
      "15095  3.402823e+38          255.0          6.0  2.041694e+39  3.402823e+38   \n",
      "17672  3.402823e+38          255.0          9.0  3.062541e+39  3.402823e+38   \n",
      "18628  3.402823e+38          255.0         10.0  3.402823e+39  3.402823e+38   \n",
      "8298   2.941716e+00           54.0          1.0  2.799062e+00  2.799062e+00   \n",
      "8440   2.678121e+00          206.0          4.0  1.020847e+39  2.552118e+38   \n",
      "\n",
      "       ...  NTLavgV_2014  NTLmaxV_2014  NTLminV_2014  NTLcfc_V_2014  \\\n",
      "16631  ...           NaN           NaN           NaN     116.000000   \n",
      "8650   ...           NaN           NaN           NaN     105.000000   \n",
      "8662   ...           NaN           NaN           NaN     111.500000   \n",
      "4073   ...           NaN           NaN           NaN      98.000000   \n",
      "15173  ...           NaN           NaN           NaN     110.333333   \n",
      "15095  ...           NaN           NaN           NaN     106.000000   \n",
      "17672  ...           NaN           NaN           NaN     119.500000   \n",
      "18628  ...           NaN           NaN           NaN     122.000000   \n",
      "8298   ...           NaN           NaN           NaN     113.000000   \n",
      "8440   ...           NaN           NaN           NaN     103.833333   \n",
      "\n",
      "       NTLctV_2015  NTLsumV_2015  NTLavgV_2015  NTLmaxV_2015  NTLminV_2015  \\\n",
      "16631          NaN           NaN           NaN           NaN           NaN   \n",
      "8650           NaN           NaN           NaN           NaN           NaN   \n",
      "8662           NaN           NaN           NaN           NaN           NaN   \n",
      "4073           NaN           NaN           NaN           NaN           NaN   \n",
      "15173          NaN           NaN           NaN           NaN           NaN   \n",
      "15095          NaN           NaN           NaN           NaN           NaN   \n",
      "17672          NaN           NaN           NaN           NaN           NaN   \n",
      "18628          NaN           NaN           NaN           NaN           NaN   \n",
      "8298           NaN           NaN           NaN           NaN           NaN   \n",
      "8440           NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2015  \n",
      "16631          101.0  \n",
      "8650           101.0  \n",
      "8662           105.5  \n",
      "4073            88.0  \n",
      "15173          106.0  \n",
      "15095          112.0  \n",
      "17672          112.5  \n",
      "18628          115.0  \n",
      "8298           106.0  \n",
      "8440           105.0  \n",
      "\n",
      "[10 rows x 115 columns]\n",
      "\n",
      "\n",
      "Finished. All years masked and assigned their nearest settlement. Sun Feb 19 10:37:56 2023\n",
      "       Sett_ID  NTLctD_1999  NTLsumD_1999  NTLavgD_1999  NTLmaxD_1999  \\\n",
      "22258   528001          3.0  6.805647e+38  2.268549e+38  3.402823e+38   \n",
      "6739    165942         19.0  5.444518e+39  2.865536e+38  3.402823e+38   \n",
      "4392    102622         26.0  8.507059e+39  3.271946e+38  3.402823e+38   \n",
      "8056    202964         23.0  5.444518e+39  2.367182e+38  3.402823e+38   \n",
      "22842   571922         49.0  1.667383e+40  3.402823e+38  3.402823e+38   \n",
      "13222   314870          NaN           NaN           NaN           NaN   \n",
      "1534     36224          2.0  6.805647e+38  3.402823e+38  3.402823e+38   \n",
      "14544   396586         49.0  1.633355e+40  3.333378e+38  3.402823e+38   \n",
      "7214    180366          8.0  2.722259e+39  3.402823e+38  3.402823e+38   \n",
      "19534   458180         13.0  4.083388e+39  3.141068e+38  3.402823e+38   \n",
      "\n",
      "       NTLminD_1999  NTLcfc_D_1999  NTLctD_2000  NTLsumD_2000  NTLavgD_2000  \\\n",
      "22258  2.807530e+00     188.000000          3.0  6.805647e+38  2.268549e+38   \n",
      "6739   2.686813e+00     223.315789         19.0  5.444518e+39  2.865536e+38   \n",
      "4392   3.094872e+00     247.192308         26.0  8.507059e+39  3.271946e+38   \n",
      "8056   2.573529e+00     194.956522         23.0  5.444518e+39  2.367182e+38   \n",
      "22842  3.402823e+38     255.000000         49.0  1.667383e+40  3.402823e+38   \n",
      "13222           NaN            NaN          NaN           NaN           NaN   \n",
      "1534   3.402823e+38     255.000000          2.0  6.805647e+38  3.402823e+38   \n",
      "14544  2.510234e+00     250.795918         49.0  1.633355e+40  3.333378e+38   \n",
      "7214   3.402823e+38     255.000000          8.0  2.722259e+39  3.402823e+38   \n",
      "19534  2.509734e+00     239.615385         13.0  4.083388e+39  3.141068e+38   \n",
      "\n",
      "       ...  NTLavgV_2014  NTLmaxV_2014  NTLminV_2014  NTLcfc_V_2014  \\\n",
      "22258  ...           NaN           NaN           NaN     106.500000   \n",
      "6739   ...           NaN           NaN           NaN      97.375000   \n",
      "4392   ...           NaN           NaN           NaN     100.000000   \n",
      "8056   ...      0.320567      0.928956      0.054024     124.666667   \n",
      "22842  ...           NaN           NaN           NaN     116.500000   \n",
      "13222  ...           NaN           NaN           NaN     101.500000   \n",
      "1534   ...           NaN           NaN           NaN      99.500000   \n",
      "14544  ...           NaN           NaN           NaN     108.666667   \n",
      "7214   ...           NaN           NaN           NaN     108.000000   \n",
      "19534  ...           NaN           NaN           NaN     110.500000   \n",
      "\n",
      "       NTLctV_2015  NTLsumV_2015  NTLavgV_2015  NTLmaxV_2015  NTLminV_2015  \\\n",
      "22258          NaN           NaN           NaN           NaN           NaN   \n",
      "6739           NaN           NaN           NaN           NaN           NaN   \n",
      "4392           NaN           NaN           NaN           NaN           NaN   \n",
      "8056          25.0      9.295749       0.37183       0.92745      0.106723   \n",
      "22842          NaN           NaN           NaN           NaN           NaN   \n",
      "13222          NaN           NaN           NaN           NaN           NaN   \n",
      "1534           NaN           NaN           NaN           NaN           NaN   \n",
      "14544          NaN           NaN           NaN           NaN           NaN   \n",
      "7214           NaN           NaN           NaN           NaN           NaN   \n",
      "19534          NaN           NaN           NaN           NaN           NaN   \n",
      "\n",
      "       NTLcfc_V_2015  \n",
      "22258      111.00000  \n",
      "6739        97.75000  \n",
      "4392        97.00000  \n",
      "8056       112.37037  \n",
      "22842      116.50000  \n",
      "13222      118.50000  \n",
      "1534        96.00000  \n",
      "14544      101.50000  \n",
      "7214       100.00000  \n",
      "19534      108.50000  \n",
      "\n",
      "[10 rows x 115 columns]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved to file. Sun Feb 19 10:37:58 2023 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "BatchZonalStats(FolderName = 'NTL', Zones=Settlements)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b80ab1eb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d3dbfb",
   "metadata": {},
   "source": [
    "## 10. FLOOD EXPOSURE BY RETURN PERIOD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07bb0569",
   "metadata": {},
   "source": [
    "### 10.1 Reclassify and set NoData values of rasters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a69ccc6d",
   "metadata": {},
   "source": [
    "##### Flood layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "id": "b96d0595",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished reclassifying FD_1in20.tif. Sun Feb 19 15:40:44 2023\n",
      "Finished reclassifying FD_1in100.tif. Sun Feb 19 15:40:47 2023\n",
      "Finished reclassifying FD_1in1000.tif. Sun Feb 19 15:40:49 2023\n",
      "\n",
      "New flood set: ['FD_1in20_binary.tif', 'FD_1in100_binary.tif', 'FD_1in1000_binary.tif']\n"
     ]
    }
   ],
   "source": [
    "InRasters = ['FD_1in20.tif', 'FD_1in100.tif', 'FD_1in1000.tif']\n",
    "BinaryRasters = []\n",
    "\n",
    "for Raster in InRasters:\n",
    "    InPath = os.path.join(ProjectFolder, 'Flood/SourceFiles', Raster)\n",
    "    OutPath = os.path.join(ProjectFolder, 'Flood/TempFiles', Raster.replace('.tif', '_binary.tif'))\n",
    "    \n",
    "    [xsize, ysize, geotransform, geoproj, Z] = readRaster(InPath)\n",
    "    \n",
    "    Z[Z<0.15] = 0\n",
    "    Z[Z>=0.15] = 1\n",
    "    \n",
    "    writeRaster(OutPath,geotransform,geoproj,Z)\n",
    "    InPath = OutPath = None\n",
    "    BinaryRasters = BinaryRasters + [Raster.replace('.tif', '_binary.tif')]\n",
    "    \n",
    "    print('Finished reclassifying %s. %s' % (Raster, time.ctime()))\n",
    "\n",
    "print('\\nNew flood set: %s' % BinaryRasters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f04af5e4",
   "metadata": {},
   "source": [
    "##### Buildup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "eebf3f5e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Simplified buildup file: Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Flood/TempFiles\\WSFE_simplified.tif\n"
     ]
    }
   ],
   "source": [
    "WSFE = 'WSFE_equalarea.tif'\n",
    "WSFEPath = os.path.join(ProjectFolder, 'Buildup', WSFE)\n",
    "OutPath = os.path.join(ProjectFolder, r'Flood/TempFiles', WSFE.replace('equalarea.tif', 'simplified.tif'))\n",
    "\n",
    "[xsize, ysize, geotransform, geoproj, Z] = readRaster(WSFEPath)\n",
    "\n",
    "np.putmask(Z, Z>0, Z-1900) # All years now converted to at most 3 digits: 85-115.\n",
    "\n",
    "writeRaster(OutPath,geotransform,geoproj,Z)\n",
    "\n",
    "print('\\nSimplified buildup file: %s' % OutPath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0692b6",
   "metadata": {},
   "source": [
    "### 10.2 Resample flood data to match buildup layer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "584bb2d4",
   "metadata": {},
   "source": [
    " - Align flood to WSFE: CRS, extent, origin, and resolution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "c548886c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#BinaryRasters = ['FD_1in20_binary.tif', 'FD_1in100_binary.tif', 'FD_1in1000_binary.tif']\n",
    "ResampledRasters = []\n",
    "\n",
    "WSFEPath = os.path.join(ProjectFolder, r'Flood/TempFiles', 'WSFE_simplified.tif') # Take new WSFE simplified file path for this, as it's what we will combine with.\n",
    "\n",
    "for Raster in BinaryRasters:\n",
    "    RasterPath = os.path.join(ProjectFolder, r'Flood/TempFiles', Raster)\n",
    "    OutPath = os.path.join(ProjectFolder, r'Flood/TempFiles', Raster.replace('_binary', '_upsample'))\n",
    "    resampleRaster(RasterPath, WSFEPath, OutPath)\n",
    "    ResampledRasters = ResampledRasters + [Raster.replace('_binary', '_upsample')]\n",
    "    \n",
    "print('Done. New flood set: %s. %s' % (ResampledRasters, time.ctime()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979d98a0",
   "metadata": {},
   "source": [
    "### 10.3 Rename flood files\n",
    "The naming conventions of the FATHOM datasets are very different from pop and NTL, \n",
    "so let's simplify the file names instead of turning the Zonal Stats script on its head.\n",
    "We will change variable names to better resemble 1 in 20, 100, and 1000 after aggregating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "id": "adfb0004",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A',\n",
       " 'B',\n",
       " 'C',\n",
       " 'E',\n",
       " 'F',\n",
       " 'G',\n",
       " 'H',\n",
       " 'I',\n",
       " 'J',\n",
       " 'K',\n",
       " 'L',\n",
       " 'M',\n",
       " 'N',\n",
       " 'O',\n",
       " 'P',\n",
       " 'Q',\n",
       " 'R',\n",
       " 'S',\n",
       " 'T',\n",
       " 'U',\n",
       " 'W',\n",
       " 'X',\n",
       " 'Y',\n",
       " 'Z']"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an alphabet that excludes letters already claimed by other variables\n",
    "ALPHA = list(string.ascii_uppercase)\n",
    "ALPHA = sorted(list(set(ALPHA) - set(['D', 'V'])))\n",
    "ALPHA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "id": "67d6518c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'A': 'FD_1in20_upsample.tif',\n",
       " 'B': 'FD_1in100_upsample.tif',\n",
       " 'C': 'FD_1in1000_upsample.tif'}"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assign a letter to each flood raster. Save to a dictionary to refer to later.\n",
    "FloodCodebook = {ALPHA[i]: ResampledRasters[i] for i in range(len(ResampledRasters))}\n",
    "FloodCodebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "id": "cb6f6607",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A_.tif', 'B_.tif', 'C_.tif']"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Rename rasters according to their dictionary key.\n",
    "for Code, Raster in FloodCodebook.items():\n",
    "    NewName = ''.join([Code, '_.tif'])\n",
    "    os.rename(f\"{'Flood'}/{'TempFiles'}/{Raster}\", f\"{'Flood'}/{'TempFiles'}/{NewName}\")\n",
    "\n",
    "RenamedRasters = [i for i in os.listdir('Flood/TempFiles') if i.endswith('_.tif')]\n",
    "RenamedRasters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dad4e00",
   "metadata": {},
   "source": [
    "### 10.5 Mask out built areas that were not flooded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "6026a3ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running for Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Flood\\TempFiles\\WSFE_simplified.tif. Tue Feb 21 22:04:09 2023\n",
      "Ran in shell. See OutFile folder to inspect results. Tue Feb 21 22:07:16 2023\n",
      "Running for Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Flood\\TempFiles\\WSFE_simplified.tif. Tue Feb 21 22:07:16 2023\n",
      "Ran in shell. See OutFile folder to inspect results. Tue Feb 21 22:10:00 2023\n",
      "Running for Q:\\GIS\\povertyequity\\urban_growth\\BurkinaFaso\\Flood\\TempFiles\\WSFE_simplified.tif. Tue Feb 21 22:10:00 2023\n",
      "Ran in shell. See OutFile folder to inspect results. Tue Feb 21 22:12:56 2023\n",
      "Done with list. New flood set: ['A_WSFE.tif', 'B_WSFE.tif', 'C_WSFE.tif']\n"
     ]
    }
   ],
   "source": [
    "RenamedRasters = ['A_.tif', 'B_.tif', 'C_.tif']\n",
    "IntersectRasters = []\n",
    "WSFEPath = os.path.join(ProjectFolder, 'Flood', 'TempFiles', 'WSFE_simplified.tif')\n",
    "    \n",
    "for Raster in RenamedRasters:\n",
    "    InPath = os.path.join(ProjectFolder, 'Flood', 'TempFiles', Raster)\n",
    "    NewFileName = Raster.replace('.tif', 'WSFE.tif')\n",
    "    OutPath = os.path.join(ProjectFolder, 'Flood', NewFileName)\n",
    "\n",
    "    calcShell(WSFEPath, InPath, OutPath, \"A*B\", OutType=\" --type=Byte\")\n",
    "    IntersectRasters = IntersectRasters + [NewFileName]\n",
    "    \n",
    "print('Done with list. New flood set: %s' % IntersectRasters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2a4250f",
   "metadata": {},
   "source": [
    "### 10.5 Flooded and non-flooded buildings summarized by settlement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8e175146",
   "metadata": {},
   "outputs": [],
   "source": [
    "Settlements = gpd.read_file(r'Results/SETTLEMENTS.gpkg', layer='SETTLEMENTS_equalarea')[['Sett_ID', 'geometry']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b201a54",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['A_WSFE.tif', 'B_WSFE.tif', 'C_WSFE.tif']\n",
      "       Sett_ID\n",
      "0            1\n",
      "1            2\n",
      "2            3\n",
      "3            4\n",
      "4            5\n",
      "...        ...\n",
      "23485   612616\n",
      "23486   612723\n",
      "23487   612934\n",
      "23488   612944\n",
      "23489   615066\n",
      "\n",
      "[23490 rows x 1 columns]\n",
      "Loading data for A_WSFE.tif. Tue Feb 21 22:18:59 2023 \n",
      "\n",
      "Creating XYZ (gdal.Translate()).\n",
      "Finished gdal.Translate() for year . Tue Feb 21 22:30:35 2023 \n",
      "\n",
      "Loaded XYZ file as a pandas dataframe. Tue Feb 21 22:46:35 2023 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "BatchZonalStats(FolderName = 'Flood', Zones = Settlements,\n",
    "                StatsWanted = ['count'], SeriesStart = 1999, SeriesEnd = 2015, VarPrefix = 'FLD')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f850b4b5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6f6ef6b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7429f6cd",
   "metadata": {},
   "source": [
    "## 11. GROWTH STATISTICS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f179e1f8",
   "metadata": {},
   "source": [
    "### 11.1 Load and prep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "06e45388",
   "metadata": {},
   "outputs": [],
   "source": [
    "# AllStudyYears = ListFromRange(1999, 2015)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "e1cedfe1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Data columns (total 2 columns):\n",
      " #   Column    Non-Null Count  Dtype \n",
      "---  ------    --------------  ----- \n",
      " 0   Sett_ID   23490 non-null  int64 \n",
      " 1   SettName  23490 non-null  object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 367.2+ KB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Data columns (total 20 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   Sett_ID   23490 non-null  int64  \n",
      " 1   year      23490 non-null  int64  \n",
      " 2   ADM_ID    23490 non-null  int64  \n",
      " 3   AREA2015  23490 non-null  float64\n",
      " 4   AREA2014  22886 non-null  float64\n",
      " 5   AREA2013  22457 non-null  float64\n",
      " 6   AREA2012  21629 non-null  float64\n",
      " 7   AREA2011  21316 non-null  float64\n",
      " 8   AREA2010  20948 non-null  float64\n",
      " 9   AREA2009  20471 non-null  float64\n",
      " 10  AREA2008  20062 non-null  float64\n",
      " 11  AREA2007  19712 non-null  float64\n",
      " 12  AREA2006  19369 non-null  float64\n",
      " 13  AREA2005  19010 non-null  float64\n",
      " 14  AREA2004  18683 non-null  float64\n",
      " 15  AREA2003  18259 non-null  float64\n",
      " 16  AREA2002  17884 non-null  float64\n",
      " 17  AREA2001  17592 non-null  float64\n",
      " 18  AREA2000  17134 non-null  float64\n",
      " 19  AREA1999  16385 non-null  float64\n",
      "dtypes: float64(17), int64(3)\n",
      "memory usage: 3.6 MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Data columns (total 33 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   Sett_ID     23490 non-null  int64  \n",
      " 1   POPct2000   23449 non-null  float64\n",
      " 2   POPsum2000  23449 non-null  float64\n",
      " 3   POPct2001   23449 non-null  float64\n",
      " 4   POPsum2001  23449 non-null  float64\n",
      " 5   POPct2002   23449 non-null  float64\n",
      " 6   POPsum2002  23449 non-null  float64\n",
      " 7   POPct2003   23449 non-null  float64\n",
      " 8   POPsum2003  23449 non-null  float64\n",
      " 9   POPct2004   23449 non-null  float64\n",
      " 10  POPsum2004  23449 non-null  float64\n",
      " 11  POPct2005   23449 non-null  float64\n",
      " 12  POPsum2005  23449 non-null  float64\n",
      " 13  POPct2006   23449 non-null  float64\n",
      " 14  POPsum2006  23449 non-null  float64\n",
      " 15  POPct2007   23449 non-null  float64\n",
      " 16  POPsum2007  23449 non-null  float64\n",
      " 17  POPct2008   23449 non-null  float64\n",
      " 18  POPsum2008  23449 non-null  float64\n",
      " 19  POPct2009   23449 non-null  float64\n",
      " 20  POPsum2009  23449 non-null  float64\n",
      " 21  POPct2010   23449 non-null  float64\n",
      " 22  POPsum2010  23449 non-null  float64\n",
      " 23  POPct2011   23449 non-null  float64\n",
      " 24  POPsum2011  23449 non-null  float64\n",
      " 25  POPct2012   23449 non-null  float64\n",
      " 26  POPsum2012  23449 non-null  float64\n",
      " 27  POPct2013   23449 non-null  float64\n",
      " 28  POPsum2013  23449 non-null  float64\n",
      " 29  POPct2014   23449 non-null  float64\n",
      " 30  POPsum2014  23449 non-null  float64\n",
      " 31  POPct2015   23449 non-null  float64\n",
      " 32  POPsum2015  23449 non-null  float64\n",
      "dtypes: float64(32), int64(1)\n",
      "memory usage: 5.9 MB\n",
      "None\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 23490 entries, 0 to 23489\n",
      "Columns: 115 entries, Sett_ID to NTLcfc_V_2015\n",
      "dtypes: float64(114), int64(1)\n",
      "memory usage: 20.6 MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "PlaceNames = pd.read_csv(os.path.join(ResultsFolder, 'PlaceNames.csv'))\n",
    "Areas = pd.read_csv(os.path.join(ResultsFolder, 'Areas1999to2015.csv'))\n",
    "Population = pd.read_csv(os.path.join(ResultsFolder, 'Population2000to2015.csv'))\n",
    "NTL = pd.read_csv(os.path.join(ResultsFolder, 'NTL1999to2015.csv'))\n",
    "\n",
    "RawValues = [PlaceNames, Areas, Population, NTL]\n",
    "\n",
    "for Dataset in RawValues:\n",
    "    if 'Unnamed: 0' in Dataset.columns:\n",
    "        Dataset.drop(columns='Unnamed: 0', inplace=True)\n",
    "    else:\n",
    "        pass\n",
    "    print(Dataset.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "07909650",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>year</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2015</th>\n",
       "      <th>AREA2014</th>\n",
       "      <th>AREA2013</th>\n",
       "      <th>AREA2012</th>\n",
       "      <th>AREA2011</th>\n",
       "      <th>AREA2010</th>\n",
       "      <th>...</th>\n",
       "      <th>NTLavgV_2014</th>\n",
       "      <th>NTLmaxV_2014</th>\n",
       "      <th>NTLminV_2014</th>\n",
       "      <th>NTLcfc_V_2014</th>\n",
       "      <th>NTLctV_2015</th>\n",
       "      <th>NTLsumV_2015</th>\n",
       "      <th>NTLavgV_2015</th>\n",
       "      <th>NTLmaxV_2015</th>\n",
       "      <th>NTLminV_2015</th>\n",
       "      <th>NTLcfc_V_2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14474</th>\n",
       "      <td>396491</td>\n",
       "      <td>Sièn</td>\n",
       "      <td>2015</td>\n",
       "      <td>311</td>\n",
       "      <td>0.203965</td>\n",
       "      <td>0.175070</td>\n",
       "      <td>0.170821</td>\n",
       "      <td>0.170821</td>\n",
       "      <td>0.170821</td>\n",
       "      <td>0.163172</td>\n",
       "      <td>...</td>\n",
       "      <td>0.199503</td>\n",
       "      <td>0.367006</td>\n",
       "      <td>0.053699</td>\n",
       "      <td>115.000000</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.904285</td>\n",
       "      <td>0.226071</td>\n",
       "      <td>0.367894</td>\n",
       "      <td>0.099993</td>\n",
       "      <td>104.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3543</th>\n",
       "      <td>87014</td>\n",
       "      <td>Gaoua</td>\n",
       "      <td>2015</td>\n",
       "      <td>96</td>\n",
       "      <td>7.829714</td>\n",
       "      <td>7.239914</td>\n",
       "      <td>7.007904</td>\n",
       "      <td>6.737650</td>\n",
       "      <td>6.537084</td>\n",
       "      <td>6.119806</td>\n",
       "      <td>...</td>\n",
       "      <td>1.778072</td>\n",
       "      <td>5.936514</td>\n",
       "      <td>0.116900</td>\n",
       "      <td>92.703704</td>\n",
       "      <td>80.0</td>\n",
       "      <td>177.543852</td>\n",
       "      <td>2.219298</td>\n",
       "      <td>5.971395</td>\n",
       "      <td>0.181498</td>\n",
       "      <td>92.320988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23336</th>\n",
       "      <td>597210</td>\n",
       "      <td>Deou</td>\n",
       "      <td>2015</td>\n",
       "      <td>65</td>\n",
       "      <td>0.328044</td>\n",
       "      <td>0.283852</td>\n",
       "      <td>0.257506</td>\n",
       "      <td>0.203965</td>\n",
       "      <td>0.201416</td>\n",
       "      <td>0.200566</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>117.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>110.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9401</th>\n",
       "      <td>215262</td>\n",
       "      <td>Kayao</td>\n",
       "      <td>2015</td>\n",
       "      <td>133</td>\n",
       "      <td>0.463171</td>\n",
       "      <td>0.455522</td>\n",
       "      <td>0.424927</td>\n",
       "      <td>0.377336</td>\n",
       "      <td>0.357789</td>\n",
       "      <td>0.350990</td>\n",
       "      <td>...</td>\n",
       "      <td>0.256452</td>\n",
       "      <td>0.699531</td>\n",
       "      <td>0.084443</td>\n",
       "      <td>107.200000</td>\n",
       "      <td>14.0</td>\n",
       "      <td>4.304194</td>\n",
       "      <td>0.307442</td>\n",
       "      <td>0.638086</td>\n",
       "      <td>0.119623</td>\n",
       "      <td>96.866667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21686</th>\n",
       "      <td>508697</td>\n",
       "      <td>Yalgo</td>\n",
       "      <td>2015</td>\n",
       "      <td>321</td>\n",
       "      <td>1.243338</td>\n",
       "      <td>1.068268</td>\n",
       "      <td>1.025775</td>\n",
       "      <td>0.973934</td>\n",
       "      <td>0.953537</td>\n",
       "      <td>0.806512</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>104.857143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>115.714286</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 167 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID SettName  year  ADM_ID  AREA2015  AREA2014  AREA2013  AREA2012  \\\n",
       "14474   396491     Sièn  2015     311  0.203965  0.175070  0.170821  0.170821   \n",
       "3543     87014    Gaoua  2015      96  7.829714  7.239914  7.007904  6.737650   \n",
       "23336   597210     Deou  2015      65  0.328044  0.283852  0.257506  0.203965   \n",
       "9401    215262    Kayao  2015     133  0.463171  0.455522  0.424927  0.377336   \n",
       "21686   508697    Yalgo  2015     321  1.243338  1.068268  1.025775  0.973934   \n",
       "\n",
       "       AREA2011  AREA2010  ...  NTLavgV_2014  NTLmaxV_2014  NTLminV_2014  \\\n",
       "14474  0.170821  0.163172  ...      0.199503      0.367006      0.053699   \n",
       "3543   6.537084  6.119806  ...      1.778072      5.936514      0.116900   \n",
       "23336  0.201416  0.200566  ...           NaN           NaN           NaN   \n",
       "9401   0.357789  0.350990  ...      0.256452      0.699531      0.084443   \n",
       "21686  0.953537  0.806512  ...           NaN           NaN           NaN   \n",
       "\n",
       "       NTLcfc_V_2014  NTLctV_2015  NTLsumV_2015  NTLavgV_2015  NTLmaxV_2015  \\\n",
       "14474     115.000000          4.0      0.904285      0.226071      0.367894   \n",
       "3543       92.703704         80.0    177.543852      2.219298      5.971395   \n",
       "23336     117.750000          NaN           NaN           NaN           NaN   \n",
       "9401      107.200000         14.0      4.304194      0.307442      0.638086   \n",
       "21686     104.857143          NaN           NaN           NaN           NaN   \n",
       "\n",
       "       NTLminV_2015  NTLcfc_V_2015  \n",
       "14474      0.099993     104.250000  \n",
       "3543       0.181498      92.320988  \n",
       "23336           NaN     110.500000  \n",
       "9401       0.119623      96.866667  \n",
       "21686           NaN     115.714286  \n",
       "\n",
       "[5 rows x 167 columns]"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "AllStats = reduce(lambda  left,right: pd.merge(left,right,on=['Sett_ID'],\n",
    "                                            how='outer'), RawValues)\n",
    "AllStats.to_csv(os.path.join(ResultsFolder, 'AllStats.csv'))\n",
    "\n",
    "AllStats[AllStats.SettName!='UNK'].sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "224573a1",
   "metadata": {},
   "source": [
    "### 11.2 Change over time of raw variables\n",
    "pch = percent change"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39152aff",
   "metadata": {},
   "source": [
    "#### Population change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "1307b9dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(Population, how = 'outer', on='Sett_ID')\n",
    "for year in AllStudyYears:\n",
    "    RawVar = ''.join(['POPsum', str(year)])\n",
    "    LagVar = ''.join(['POPsum', str(year-1)])\n",
    "    NewVar = ''.join(['POPpch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "e720c495",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName', 'POPpch2001', 'POPpch2002', 'POPpch2003',\n",
       "       'POPpch2004', 'POPpch2005', 'POPpch2006', 'POPpch2007', 'POPpch2008',\n",
       "       'POPpch2009', 'POPpch2010', 'POPpch2011', 'POPpch2012', 'POPpch2013',\n",
       "       'POPpch2014', 'POPpch2015'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|ADM_ID|pch')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "f93e099b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>year</th>\n",
       "      <th>ADM_ID</th>\n",
       "      <th>AREA2015</th>\n",
       "      <th>AREA2014</th>\n",
       "      <th>AREA2013</th>\n",
       "      <th>AREA2012</th>\n",
       "      <th>AREA2011</th>\n",
       "      <th>AREA2010</th>\n",
       "      <th>...</th>\n",
       "      <th>POPpch2006</th>\n",
       "      <th>POPpch2007</th>\n",
       "      <th>POPpch2008</th>\n",
       "      <th>POPpch2009</th>\n",
       "      <th>POPpch2010</th>\n",
       "      <th>POPpch2011</th>\n",
       "      <th>POPpch2012</th>\n",
       "      <th>POPpch2013</th>\n",
       "      <th>POPpch2014</th>\n",
       "      <th>POPpch2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9910</th>\n",
       "      <td>227047</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>338.023014</td>\n",
       "      <td>326.382551</td>\n",
       "      <td>318.261337</td>\n",
       "      <td>310.990827</td>\n",
       "      <td>304.469890</td>\n",
       "      <td>...</td>\n",
       "      <td>0.074921</td>\n",
       "      <td>0.063120</td>\n",
       "      <td>0.071251</td>\n",
       "      <td>0.070549</td>\n",
       "      <td>0.064425</td>\n",
       "      <td>0.068094</td>\n",
       "      <td>0.062830</td>\n",
       "      <td>0.063821</td>\n",
       "      <td>0.062000</td>\n",
       "      <td>0.065201</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2177</th>\n",
       "      <td>51196</td>\n",
       "      <td>Bobo-Dioulasso</td>\n",
       "      <td>2015</td>\n",
       "      <td>37</td>\n",
       "      <td>88.690864</td>\n",
       "      <td>87.474722</td>\n",
       "      <td>86.498238</td>\n",
       "      <td>84.906460</td>\n",
       "      <td>82.475025</td>\n",
       "      <td>80.240756</td>\n",
       "      <td>...</td>\n",
       "      <td>0.119239</td>\n",
       "      <td>0.017022</td>\n",
       "      <td>0.024202</td>\n",
       "      <td>0.057145</td>\n",
       "      <td>0.041187</td>\n",
       "      <td>0.146031</td>\n",
       "      <td>-0.016292</td>\n",
       "      <td>0.077231</td>\n",
       "      <td>0.076259</td>\n",
       "      <td>0.013518</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8057</th>\n",
       "      <td>202965</td>\n",
       "      <td>Koudougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>157</td>\n",
       "      <td>30.684014</td>\n",
       "      <td>30.157103</td>\n",
       "      <td>29.583451</td>\n",
       "      <td>28.301020</td>\n",
       "      <td>27.843798</td>\n",
       "      <td>27.413772</td>\n",
       "      <td>...</td>\n",
       "      <td>0.061063</td>\n",
       "      <td>0.036937</td>\n",
       "      <td>0.005316</td>\n",
       "      <td>-0.003620</td>\n",
       "      <td>0.046503</td>\n",
       "      <td>0.045571</td>\n",
       "      <td>-0.047556</td>\n",
       "      <td>0.062258</td>\n",
       "      <td>0.065706</td>\n",
       "      <td>0.004961</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Banfora</td>\n",
       "      <td>2015</td>\n",
       "      <td>14</td>\n",
       "      <td>16.495685</td>\n",
       "      <td>16.022316</td>\n",
       "      <td>15.331384</td>\n",
       "      <td>14.787476</td>\n",
       "      <td>14.615806</td>\n",
       "      <td>14.377846</td>\n",
       "      <td>...</td>\n",
       "      <td>0.048915</td>\n",
       "      <td>0.067632</td>\n",
       "      <td>-0.037615</td>\n",
       "      <td>0.066185</td>\n",
       "      <td>0.000993</td>\n",
       "      <td>0.158843</td>\n",
       "      <td>-0.093080</td>\n",
       "      <td>0.095768</td>\n",
       "      <td>0.088328</td>\n",
       "      <td>-0.003954</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18503</th>\n",
       "      <td>456905</td>\n",
       "      <td>Ouahigouya</td>\n",
       "      <td>2015</td>\n",
       "      <td>219</td>\n",
       "      <td>13.275584</td>\n",
       "      <td>12.025448</td>\n",
       "      <td>10.522054</td>\n",
       "      <td>9.225176</td>\n",
       "      <td>9.060304</td>\n",
       "      <td>8.580136</td>\n",
       "      <td>...</td>\n",
       "      <td>0.055096</td>\n",
       "      <td>-0.005316</td>\n",
       "      <td>0.087090</td>\n",
       "      <td>0.000485</td>\n",
       "      <td>0.009797</td>\n",
       "      <td>0.069090</td>\n",
       "      <td>-0.077206</td>\n",
       "      <td>0.126531</td>\n",
       "      <td>0.090186</td>\n",
       "      <td>-0.008332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5552</th>\n",
       "      <td>137698</td>\n",
       "      <td>Dedougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>64</td>\n",
       "      <td>12.117232</td>\n",
       "      <td>11.673608</td>\n",
       "      <td>11.322618</td>\n",
       "      <td>10.958880</td>\n",
       "      <td>10.829702</td>\n",
       "      <td>10.616388</td>\n",
       "      <td>...</td>\n",
       "      <td>0.185645</td>\n",
       "      <td>-0.079261</td>\n",
       "      <td>0.066683</td>\n",
       "      <td>-0.036713</td>\n",
       "      <td>0.061705</td>\n",
       "      <td>0.154599</td>\n",
       "      <td>-0.029161</td>\n",
       "      <td>-0.006150</td>\n",
       "      <td>0.067749</td>\n",
       "      <td>0.155712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10721</th>\n",
       "      <td>247085</td>\n",
       "      <td>Tenkodogo</td>\n",
       "      <td>2015</td>\n",
       "      <td>296</td>\n",
       "      <td>11.164545</td>\n",
       "      <td>11.025168</td>\n",
       "      <td>10.938483</td>\n",
       "      <td>10.821203</td>\n",
       "      <td>10.260299</td>\n",
       "      <td>9.979847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.090468</td>\n",
       "      <td>0.043719</td>\n",
       "      <td>0.003318</td>\n",
       "      <td>0.053585</td>\n",
       "      <td>-0.017382</td>\n",
       "      <td>0.010022</td>\n",
       "      <td>-0.103379</td>\n",
       "      <td>0.030291</td>\n",
       "      <td>0.122845</td>\n",
       "      <td>0.065079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12674</th>\n",
       "      <td>309529</td>\n",
       "      <td>Pouytenga</td>\n",
       "      <td>2015</td>\n",
       "      <td>249</td>\n",
       "      <td>10.984375</td>\n",
       "      <td>10.879843</td>\n",
       "      <td>10.762563</td>\n",
       "      <td>10.671629</td>\n",
       "      <td>10.528853</td>\n",
       "      <td>10.370780</td>\n",
       "      <td>...</td>\n",
       "      <td>0.058226</td>\n",
       "      <td>0.056197</td>\n",
       "      <td>0.085240</td>\n",
       "      <td>-0.011247</td>\n",
       "      <td>0.018975</td>\n",
       "      <td>0.077712</td>\n",
       "      <td>-0.025278</td>\n",
       "      <td>0.096485</td>\n",
       "      <td>0.123198</td>\n",
       "      <td>0.028576</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13277</th>\n",
       "      <td>318632</td>\n",
       "      <td>Fada N'gourma</td>\n",
       "      <td>2015</td>\n",
       "      <td>87</td>\n",
       "      <td>10.748116</td>\n",
       "      <td>10.528003</td>\n",
       "      <td>10.369930</td>\n",
       "      <td>10.135370</td>\n",
       "      <td>9.806476</td>\n",
       "      <td>9.706194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.034875</td>\n",
       "      <td>0.016980</td>\n",
       "      <td>0.096244</td>\n",
       "      <td>0.063301</td>\n",
       "      <td>0.032086</td>\n",
       "      <td>0.081722</td>\n",
       "      <td>-0.028733</td>\n",
       "      <td>0.051997</td>\n",
       "      <td>0.034627</td>\n",
       "      <td>0.092794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4339</th>\n",
       "      <td>102549</td>\n",
       "      <td>Hounde</td>\n",
       "      <td>2015</td>\n",
       "      <td>116</td>\n",
       "      <td>9.152938</td>\n",
       "      <td>8.733959</td>\n",
       "      <td>8.262290</td>\n",
       "      <td>7.708185</td>\n",
       "      <td>7.514418</td>\n",
       "      <td>7.334248</td>\n",
       "      <td>...</td>\n",
       "      <td>0.073129</td>\n",
       "      <td>0.026597</td>\n",
       "      <td>0.019874</td>\n",
       "      <td>0.061333</td>\n",
       "      <td>0.071136</td>\n",
       "      <td>0.001144</td>\n",
       "      <td>-0.088944</td>\n",
       "      <td>0.363905</td>\n",
       "      <td>0.133101</td>\n",
       "      <td>0.032521</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 182 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID        SettName  year  ADM_ID    AREA2015    AREA2014  \\\n",
       "9910    227047     Ouagadougou  2015     147  346.334596  338.023014   \n",
       "2177     51196  Bobo-Dioulasso  2015      37   88.690864   87.474722   \n",
       "8057    202965       Koudougou  2015     157   30.684014   30.157103   \n",
       "7            8         Banfora  2015      14   16.495685   16.022316   \n",
       "18503   456905      Ouahigouya  2015     219   13.275584   12.025448   \n",
       "5552    137698        Dedougou  2015      64   12.117232   11.673608   \n",
       "10721   247085       Tenkodogo  2015     296   11.164545   11.025168   \n",
       "12674   309529       Pouytenga  2015     249   10.984375   10.879843   \n",
       "13277   318632   Fada N'gourma  2015      87   10.748116   10.528003   \n",
       "4339    102549          Hounde  2015     116    9.152938    8.733959   \n",
       "\n",
       "         AREA2013    AREA2012    AREA2011    AREA2010  ...  POPpch2006  \\\n",
       "9910   326.382551  318.261337  310.990827  304.469890  ...    0.074921   \n",
       "2177    86.498238   84.906460   82.475025   80.240756  ...    0.119239   \n",
       "8057    29.583451   28.301020   27.843798   27.413772  ...    0.061063   \n",
       "7       15.331384   14.787476   14.615806   14.377846  ...    0.048915   \n",
       "18503   10.522054    9.225176    9.060304    8.580136  ...    0.055096   \n",
       "5552    11.322618   10.958880   10.829702   10.616388  ...    0.185645   \n",
       "10721   10.938483   10.821203   10.260299    9.979847  ...    0.090468   \n",
       "12674   10.762563   10.671629   10.528853   10.370780  ...    0.058226   \n",
       "13277   10.369930   10.135370    9.806476    9.706194  ...    0.034875   \n",
       "4339     8.262290    7.708185    7.514418    7.334248  ...    0.073129   \n",
       "\n",
       "       POPpch2007  POPpch2008  POPpch2009  POPpch2010  POPpch2011  POPpch2012  \\\n",
       "9910     0.063120    0.071251    0.070549    0.064425    0.068094    0.062830   \n",
       "2177     0.017022    0.024202    0.057145    0.041187    0.146031   -0.016292   \n",
       "8057     0.036937    0.005316   -0.003620    0.046503    0.045571   -0.047556   \n",
       "7        0.067632   -0.037615    0.066185    0.000993    0.158843   -0.093080   \n",
       "18503   -0.005316    0.087090    0.000485    0.009797    0.069090   -0.077206   \n",
       "5552    -0.079261    0.066683   -0.036713    0.061705    0.154599   -0.029161   \n",
       "10721    0.043719    0.003318    0.053585   -0.017382    0.010022   -0.103379   \n",
       "12674    0.056197    0.085240   -0.011247    0.018975    0.077712   -0.025278   \n",
       "13277    0.016980    0.096244    0.063301    0.032086    0.081722   -0.028733   \n",
       "4339     0.026597    0.019874    0.061333    0.071136    0.001144   -0.088944   \n",
       "\n",
       "       POPpch2013  POPpch2014  POPpch2015  \n",
       "9910     0.063821    0.062000    0.065201  \n",
       "2177     0.077231    0.076259    0.013518  \n",
       "8057     0.062258    0.065706    0.004961  \n",
       "7        0.095768    0.088328   -0.003954  \n",
       "18503    0.126531    0.090186   -0.008332  \n",
       "5552    -0.006150    0.067749    0.155712  \n",
       "10721    0.030291    0.122845    0.065079  \n",
       "12674    0.096485    0.123198    0.028576  \n",
       "13277    0.051997    0.034627    0.092794  \n",
       "4339     0.363905    0.133101    0.032521  \n",
       "\n",
       "[10 rows x 182 columns]"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(ResultsFolder, 'PopChange.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87709ba2",
   "metadata": {},
   "source": [
    "#### Area change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "0deb53d5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(Areas, how = 'outer', on='Sett_ID')\n",
    "for year in AllStudyYears:\n",
    "    RawVar = ''.join(['AREA', str(year)])\n",
    "    LagVar = ''.join(['AREA', str(year-1)])\n",
    "    NewVar = ''.join(['AREApch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "1569a1e8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName', 'ADM_ID', 'AREApch2000', 'AREApch2001',\n",
       "       'AREApch2002', 'AREApch2003', 'AREApch2004', 'AREApch2005',\n",
       "       'AREApch2006', 'AREApch2007', 'AREApch2008', 'AREApch2009',\n",
       "       'AREApch2010', 'AREApch2011', 'AREApch2012', 'AREApch2013',\n",
       "       'AREApch2014', 'AREApch2015'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|ADM_ID|pch')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "743cb7ec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>year</th>\n",
       "      <th>ADM_ID_x</th>\n",
       "      <th>AREA2015</th>\n",
       "      <th>AREA2014</th>\n",
       "      <th>AREA2013</th>\n",
       "      <th>AREA2012</th>\n",
       "      <th>AREA2011</th>\n",
       "      <th>AREA2010</th>\n",
       "      <th>...</th>\n",
       "      <th>AREApch2006</th>\n",
       "      <th>AREApch2007</th>\n",
       "      <th>AREApch2008</th>\n",
       "      <th>AREApch2009</th>\n",
       "      <th>AREApch2010</th>\n",
       "      <th>AREApch2011</th>\n",
       "      <th>AREApch2012</th>\n",
       "      <th>AREApch2013</th>\n",
       "      <th>AREApch2014</th>\n",
       "      <th>AREApch2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9910</th>\n",
       "      <td>227047</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>338.023014</td>\n",
       "      <td>326.382551</td>\n",
       "      <td>318.261337</td>\n",
       "      <td>310.990827</td>\n",
       "      <td>304.469890</td>\n",
       "      <td>...</td>\n",
       "      <td>0.026976</td>\n",
       "      <td>0.028081</td>\n",
       "      <td>0.017805</td>\n",
       "      <td>0.023416</td>\n",
       "      <td>0.047535</td>\n",
       "      <td>0.021417</td>\n",
       "      <td>0.023379</td>\n",
       "      <td>0.025517</td>\n",
       "      <td>0.035665</td>\n",
       "      <td>0.024589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2177</th>\n",
       "      <td>51196</td>\n",
       "      <td>Bobo-Dioulasso</td>\n",
       "      <td>2015</td>\n",
       "      <td>37</td>\n",
       "      <td>88.690864</td>\n",
       "      <td>87.474722</td>\n",
       "      <td>86.498238</td>\n",
       "      <td>84.906460</td>\n",
       "      <td>82.475025</td>\n",
       "      <td>80.240756</td>\n",
       "      <td>...</td>\n",
       "      <td>0.016859</td>\n",
       "      <td>0.020165</td>\n",
       "      <td>0.021398</td>\n",
       "      <td>0.021354</td>\n",
       "      <td>0.039526</td>\n",
       "      <td>0.027845</td>\n",
       "      <td>0.029481</td>\n",
       "      <td>0.018747</td>\n",
       "      <td>0.011289</td>\n",
       "      <td>0.013903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8057</th>\n",
       "      <td>202965</td>\n",
       "      <td>Koudougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>157</td>\n",
       "      <td>30.684014</td>\n",
       "      <td>30.157103</td>\n",
       "      <td>29.583451</td>\n",
       "      <td>28.301020</td>\n",
       "      <td>27.843798</td>\n",
       "      <td>27.413772</td>\n",
       "      <td>...</td>\n",
       "      <td>0.019236</td>\n",
       "      <td>0.012149</td>\n",
       "      <td>0.023481</td>\n",
       "      <td>0.015166</td>\n",
       "      <td>0.020985</td>\n",
       "      <td>0.015687</td>\n",
       "      <td>0.016421</td>\n",
       "      <td>0.045314</td>\n",
       "      <td>0.019391</td>\n",
       "      <td>0.017472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Banfora</td>\n",
       "      <td>2015</td>\n",
       "      <td>14</td>\n",
       "      <td>16.495685</td>\n",
       "      <td>16.022316</td>\n",
       "      <td>15.331384</td>\n",
       "      <td>14.787476</td>\n",
       "      <td>14.615806</td>\n",
       "      <td>14.377846</td>\n",
       "      <td>...</td>\n",
       "      <td>0.033342</td>\n",
       "      <td>0.036258</td>\n",
       "      <td>0.020352</td>\n",
       "      <td>0.040081</td>\n",
       "      <td>0.023472</td>\n",
       "      <td>0.016550</td>\n",
       "      <td>0.011746</td>\n",
       "      <td>0.036782</td>\n",
       "      <td>0.045067</td>\n",
       "      <td>0.029544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18503</th>\n",
       "      <td>456905</td>\n",
       "      <td>Ouahigouya</td>\n",
       "      <td>2015</td>\n",
       "      <td>219</td>\n",
       "      <td>13.275584</td>\n",
       "      <td>12.025448</td>\n",
       "      <td>10.522054</td>\n",
       "      <td>9.225176</td>\n",
       "      <td>9.060304</td>\n",
       "      <td>8.580136</td>\n",
       "      <td>...</td>\n",
       "      <td>0.055474</td>\n",
       "      <td>0.073759</td>\n",
       "      <td>0.050603</td>\n",
       "      <td>0.070560</td>\n",
       "      <td>0.061285</td>\n",
       "      <td>0.055963</td>\n",
       "      <td>0.018197</td>\n",
       "      <td>0.140580</td>\n",
       "      <td>0.142880</td>\n",
       "      <td>0.103958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5552</th>\n",
       "      <td>137698</td>\n",
       "      <td>Dedougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>64</td>\n",
       "      <td>12.117232</td>\n",
       "      <td>11.673608</td>\n",
       "      <td>11.322618</td>\n",
       "      <td>10.958880</td>\n",
       "      <td>10.829702</td>\n",
       "      <td>10.616388</td>\n",
       "      <td>...</td>\n",
       "      <td>0.018177</td>\n",
       "      <td>0.039577</td>\n",
       "      <td>0.031983</td>\n",
       "      <td>0.020954</td>\n",
       "      <td>0.077268</td>\n",
       "      <td>0.020093</td>\n",
       "      <td>0.011928</td>\n",
       "      <td>0.033191</td>\n",
       "      <td>0.030999</td>\n",
       "      <td>0.038002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10721</th>\n",
       "      <td>247085</td>\n",
       "      <td>Tenkodogo</td>\n",
       "      <td>2015</td>\n",
       "      <td>296</td>\n",
       "      <td>11.164545</td>\n",
       "      <td>11.025168</td>\n",
       "      <td>10.938483</td>\n",
       "      <td>10.821203</td>\n",
       "      <td>10.260299</td>\n",
       "      <td>9.979847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017887</td>\n",
       "      <td>0.023267</td>\n",
       "      <td>0.024849</td>\n",
       "      <td>0.046901</td>\n",
       "      <td>0.050076</td>\n",
       "      <td>0.028102</td>\n",
       "      <td>0.054667</td>\n",
       "      <td>0.010838</td>\n",
       "      <td>0.007925</td>\n",
       "      <td>0.012642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12674</th>\n",
       "      <td>309529</td>\n",
       "      <td>Pouytenga</td>\n",
       "      <td>2015</td>\n",
       "      <td>249</td>\n",
       "      <td>10.984375</td>\n",
       "      <td>10.879843</td>\n",
       "      <td>10.762563</td>\n",
       "      <td>10.671629</td>\n",
       "      <td>10.528853</td>\n",
       "      <td>10.370780</td>\n",
       "      <td>...</td>\n",
       "      <td>0.009548</td>\n",
       "      <td>0.009921</td>\n",
       "      <td>0.042692</td>\n",
       "      <td>0.036189</td>\n",
       "      <td>0.036965</td>\n",
       "      <td>0.015242</td>\n",
       "      <td>0.013560</td>\n",
       "      <td>0.008521</td>\n",
       "      <td>0.010897</td>\n",
       "      <td>0.009608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13277</th>\n",
       "      <td>318632</td>\n",
       "      <td>Fada N'gourma</td>\n",
       "      <td>2015</td>\n",
       "      <td>87</td>\n",
       "      <td>10.748116</td>\n",
       "      <td>10.528003</td>\n",
       "      <td>10.369930</td>\n",
       "      <td>10.135370</td>\n",
       "      <td>9.806476</td>\n",
       "      <td>9.706194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.016576</td>\n",
       "      <td>0.041111</td>\n",
       "      <td>0.016327</td>\n",
       "      <td>0.026431</td>\n",
       "      <td>0.039217</td>\n",
       "      <td>0.010332</td>\n",
       "      <td>0.033538</td>\n",
       "      <td>0.023143</td>\n",
       "      <td>0.015243</td>\n",
       "      <td>0.020907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4339</th>\n",
       "      <td>102549</td>\n",
       "      <td>Hounde</td>\n",
       "      <td>2015</td>\n",
       "      <td>116</td>\n",
       "      <td>9.152938</td>\n",
       "      <td>8.733959</td>\n",
       "      <td>8.262290</td>\n",
       "      <td>7.708185</td>\n",
       "      <td>7.514418</td>\n",
       "      <td>7.334248</td>\n",
       "      <td>...</td>\n",
       "      <td>0.065906</td>\n",
       "      <td>0.109975</td>\n",
       "      <td>0.050319</td>\n",
       "      <td>0.090958</td>\n",
       "      <td>0.067541</td>\n",
       "      <td>0.024565</td>\n",
       "      <td>0.025786</td>\n",
       "      <td>0.071885</td>\n",
       "      <td>0.057087</td>\n",
       "      <td>0.047971</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 199 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID        SettName  year  ADM_ID_x    AREA2015    AREA2014  \\\n",
       "9910    227047     Ouagadougou  2015       147  346.334596  338.023014   \n",
       "2177     51196  Bobo-Dioulasso  2015        37   88.690864   87.474722   \n",
       "8057    202965       Koudougou  2015       157   30.684014   30.157103   \n",
       "7            8         Banfora  2015        14   16.495685   16.022316   \n",
       "18503   456905      Ouahigouya  2015       219   13.275584   12.025448   \n",
       "5552    137698        Dedougou  2015        64   12.117232   11.673608   \n",
       "10721   247085       Tenkodogo  2015       296   11.164545   11.025168   \n",
       "12674   309529       Pouytenga  2015       249   10.984375   10.879843   \n",
       "13277   318632   Fada N'gourma  2015        87   10.748116   10.528003   \n",
       "4339    102549          Hounde  2015       116    9.152938    8.733959   \n",
       "\n",
       "         AREA2013    AREA2012    AREA2011    AREA2010  ...  AREApch2006  \\\n",
       "9910   326.382551  318.261337  310.990827  304.469890  ...     0.026976   \n",
       "2177    86.498238   84.906460   82.475025   80.240756  ...     0.016859   \n",
       "8057    29.583451   28.301020   27.843798   27.413772  ...     0.019236   \n",
       "7       15.331384   14.787476   14.615806   14.377846  ...     0.033342   \n",
       "18503   10.522054    9.225176    9.060304    8.580136  ...     0.055474   \n",
       "5552    11.322618   10.958880   10.829702   10.616388  ...     0.018177   \n",
       "10721   10.938483   10.821203   10.260299    9.979847  ...     0.017887   \n",
       "12674   10.762563   10.671629   10.528853   10.370780  ...     0.009548   \n",
       "13277   10.369930   10.135370    9.806476    9.706194  ...     0.016576   \n",
       "4339     8.262290    7.708185    7.514418    7.334248  ...     0.065906   \n",
       "\n",
       "       AREApch2007  AREApch2008  AREApch2009  AREApch2010  AREApch2011  \\\n",
       "9910      0.028081     0.017805     0.023416     0.047535     0.021417   \n",
       "2177      0.020165     0.021398     0.021354     0.039526     0.027845   \n",
       "8057      0.012149     0.023481     0.015166     0.020985     0.015687   \n",
       "7         0.036258     0.020352     0.040081     0.023472     0.016550   \n",
       "18503     0.073759     0.050603     0.070560     0.061285     0.055963   \n",
       "5552      0.039577     0.031983     0.020954     0.077268     0.020093   \n",
       "10721     0.023267     0.024849     0.046901     0.050076     0.028102   \n",
       "12674     0.009921     0.042692     0.036189     0.036965     0.015242   \n",
       "13277     0.041111     0.016327     0.026431     0.039217     0.010332   \n",
       "4339      0.109975     0.050319     0.090958     0.067541     0.024565   \n",
       "\n",
       "       AREApch2012  AREApch2013  AREApch2014  AREApch2015  \n",
       "9910      0.023379     0.025517     0.035665     0.024589  \n",
       "2177      0.029481     0.018747     0.011289     0.013903  \n",
       "8057      0.016421     0.045314     0.019391     0.017472  \n",
       "7         0.011746     0.036782     0.045067     0.029544  \n",
       "18503     0.018197     0.140580     0.142880     0.103958  \n",
       "5552      0.011928     0.033191     0.030999     0.038002  \n",
       "10721     0.054667     0.010838     0.007925     0.012642  \n",
       "12674     0.013560     0.008521     0.010897     0.009608  \n",
       "13277     0.033538     0.023143     0.015243     0.020907  \n",
       "4339      0.025786     0.071885     0.057087     0.047971  \n",
       "\n",
       "[10 rows x 199 columns]"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(ResultsFolder, 'AreaChange.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eacb6568",
   "metadata": {},
   "source": [
    "#### NTL change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "ff131334",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(NTL, how = 'outer', on='Sett_ID')\n",
    "Sensors = ['D_', 'V_']\n",
    "Methods = ['sum', 'avg', 'max']\n",
    "\n",
    "for year in AllStudyYears:\n",
    "    for Sensor in Sensors:\n",
    "        for agg in Methods:\n",
    "            RawVar = ''.join(['NTL', agg, Sensor, str(year)])\n",
    "            LagVar = ''.join(['NTL', agg, Sensor, str(year-1)])\n",
    "            NewVar = ''.join(['NTL', agg, '_pch', Sensor, str(year)])\n",
    "            if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "                Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "            else:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "1104d70a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName', 'NTLsum_pchD_2000', 'NTLavg_pchD_2000',\n",
       "       'NTLmax_pchD_2000', 'NTLsum_pchD_2001', 'NTLavg_pchD_2001',\n",
       "       'NTLmax_pchD_2001', 'NTLsum_pchD_2002', 'NTLavg_pchD_2002',\n",
       "       'NTLmax_pchD_2002', 'NTLsum_pchD_2003', 'NTLavg_pchD_2003',\n",
       "       'NTLmax_pchD_2003', 'NTLsum_pchD_2004', 'NTLavg_pchD_2004',\n",
       "       'NTLmax_pchD_2004', 'NTLsum_pchD_2005', 'NTLavg_pchD_2005',\n",
       "       'NTLmax_pchD_2005', 'NTLsum_pchD_2006', 'NTLavg_pchD_2006',\n",
       "       'NTLmax_pchD_2006', 'NTLsum_pchD_2007', 'NTLavg_pchD_2007',\n",
       "       'NTLmax_pchD_2007', 'NTLsum_pchD_2008', 'NTLavg_pchD_2008',\n",
       "       'NTLmax_pchD_2008', 'NTLsum_pchD_2009', 'NTLavg_pchD_2009',\n",
       "       'NTLmax_pchD_2009', 'NTLsum_pchD_2010', 'NTLavg_pchD_2010',\n",
       "       'NTLmax_pchD_2010', 'NTLsum_pchD_2011', 'NTLavg_pchD_2011',\n",
       "       'NTLmax_pchD_2011', 'NTLsum_pchD_2012', 'NTLavg_pchD_2012',\n",
       "       'NTLmax_pchD_2012', 'NTLsum_pchD_2013', 'NTLavg_pchD_2013',\n",
       "       'NTLmax_pchD_2013', 'NTLsum_pchV_2013', 'NTLavg_pchV_2013',\n",
       "       'NTLmax_pchV_2013', 'NTLsum_pchV_2014', 'NTLavg_pchV_2014',\n",
       "       'NTLmax_pchV_2014', 'NTLsum_pchV_2015', 'NTLavg_pchV_2015',\n",
       "       'NTLmax_pchV_2015'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|pch')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "76fd27f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>year</th>\n",
       "      <th>ADM_ID_x</th>\n",
       "      <th>AREA2015</th>\n",
       "      <th>AREA2014</th>\n",
       "      <th>AREA2013</th>\n",
       "      <th>AREA2012</th>\n",
       "      <th>AREA2011</th>\n",
       "      <th>AREA2010</th>\n",
       "      <th>...</th>\n",
       "      <th>NTLmax_pchD_2013</th>\n",
       "      <th>NTLsum_pchV_2013</th>\n",
       "      <th>NTLavg_pchV_2013</th>\n",
       "      <th>NTLmax_pchV_2013</th>\n",
       "      <th>NTLsum_pchV_2014</th>\n",
       "      <th>NTLavg_pchV_2014</th>\n",
       "      <th>NTLmax_pchV_2014</th>\n",
       "      <th>NTLsum_pchV_2015</th>\n",
       "      <th>NTLavg_pchV_2015</th>\n",
       "      <th>NTLmax_pchV_2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9910</th>\n",
       "      <td>227047</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>338.023014</td>\n",
       "      <td>326.382551</td>\n",
       "      <td>318.261337</td>\n",
       "      <td>310.990827</td>\n",
       "      <td>304.469890</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.105390</td>\n",
       "      <td>0.046837</td>\n",
       "      <td>-0.036419</td>\n",
       "      <td>0.019402</td>\n",
       "      <td>-0.018029</td>\n",
       "      <td>0.093730</td>\n",
       "      <td>-0.006311</td>\n",
       "      <td>-0.057119</td>\n",
       "      <td>0.156668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2177</th>\n",
       "      <td>51196</td>\n",
       "      <td>Bobo-Dioulasso</td>\n",
       "      <td>2015</td>\n",
       "      <td>37</td>\n",
       "      <td>88.690864</td>\n",
       "      <td>87.474722</td>\n",
       "      <td>86.498238</td>\n",
       "      <td>84.906460</td>\n",
       "      <td>82.475025</td>\n",
       "      <td>80.240756</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.097519</td>\n",
       "      <td>0.070837</td>\n",
       "      <td>-0.015637</td>\n",
       "      <td>-0.033550</td>\n",
       "      <td>-0.042858</td>\n",
       "      <td>-0.107115</td>\n",
       "      <td>0.074999</td>\n",
       "      <td>0.028763</td>\n",
       "      <td>-0.079833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8057</th>\n",
       "      <td>202965</td>\n",
       "      <td>Koudougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>157</td>\n",
       "      <td>30.684014</td>\n",
       "      <td>30.157103</td>\n",
       "      <td>29.583451</td>\n",
       "      <td>28.301020</td>\n",
       "      <td>27.843798</td>\n",
       "      <td>27.413772</td>\n",
       "      <td>...</td>\n",
       "      <td>0.202622</td>\n",
       "      <td>0.540515</td>\n",
       "      <td>0.368111</td>\n",
       "      <td>1.384203</td>\n",
       "      <td>-0.128261</td>\n",
       "      <td>-0.118716</td>\n",
       "      <td>-0.243544</td>\n",
       "      <td>-0.056475</td>\n",
       "      <td>-0.059906</td>\n",
       "      <td>-0.317351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Banfora</td>\n",
       "      <td>2015</td>\n",
       "      <td>14</td>\n",
       "      <td>16.495685</td>\n",
       "      <td>16.022316</td>\n",
       "      <td>15.331384</td>\n",
       "      <td>14.787476</td>\n",
       "      <td>14.615806</td>\n",
       "      <td>14.377846</td>\n",
       "      <td>...</td>\n",
       "      <td>0.003737</td>\n",
       "      <td>0.130527</td>\n",
       "      <td>0.139091</td>\n",
       "      <td>0.143354</td>\n",
       "      <td>-0.069417</td>\n",
       "      <td>-0.096787</td>\n",
       "      <td>-0.154211</td>\n",
       "      <td>0.265583</td>\n",
       "      <td>0.265583</td>\n",
       "      <td>-0.061563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18503</th>\n",
       "      <td>456905</td>\n",
       "      <td>Ouahigouya</td>\n",
       "      <td>2015</td>\n",
       "      <td>219</td>\n",
       "      <td>13.275584</td>\n",
       "      <td>12.025448</td>\n",
       "      <td>10.522054</td>\n",
       "      <td>9.225176</td>\n",
       "      <td>9.060304</td>\n",
       "      <td>8.580136</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.026280</td>\n",
       "      <td>0.153354</td>\n",
       "      <td>0.098056</td>\n",
       "      <td>0.050841</td>\n",
       "      <td>0.062930</td>\n",
       "      <td>0.034585</td>\n",
       "      <td>0.086015</td>\n",
       "      <td>-0.124337</td>\n",
       "      <td>-0.130137</td>\n",
       "      <td>-0.248345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5552</th>\n",
       "      <td>137698</td>\n",
       "      <td>Dedougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>64</td>\n",
       "      <td>12.117232</td>\n",
       "      <td>11.673608</td>\n",
       "      <td>11.322618</td>\n",
       "      <td>10.958880</td>\n",
       "      <td>10.829702</td>\n",
       "      <td>10.616388</td>\n",
       "      <td>...</td>\n",
       "      <td>0.011141</td>\n",
       "      <td>-0.072531</td>\n",
       "      <td>-0.082294</td>\n",
       "      <td>-0.499691</td>\n",
       "      <td>0.475158</td>\n",
       "      <td>0.401400</td>\n",
       "      <td>0.176559</td>\n",
       "      <td>0.672705</td>\n",
       "      <td>0.371069</td>\n",
       "      <td>0.275277</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10721</th>\n",
       "      <td>247085</td>\n",
       "      <td>Tenkodogo</td>\n",
       "      <td>2015</td>\n",
       "      <td>296</td>\n",
       "      <td>11.164545</td>\n",
       "      <td>11.025168</td>\n",
       "      <td>10.938483</td>\n",
       "      <td>10.821203</td>\n",
       "      <td>10.260299</td>\n",
       "      <td>9.979847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.043578</td>\n",
       "      <td>0.110538</td>\n",
       "      <td>0.100796</td>\n",
       "      <td>-0.037817</td>\n",
       "      <td>-0.015631</td>\n",
       "      <td>-0.080180</td>\n",
       "      <td>-0.188068</td>\n",
       "      <td>-0.078570</td>\n",
       "      <td>-0.063213</td>\n",
       "      <td>-0.304051</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12674</th>\n",
       "      <td>309529</td>\n",
       "      <td>Pouytenga</td>\n",
       "      <td>2015</td>\n",
       "      <td>249</td>\n",
       "      <td>10.984375</td>\n",
       "      <td>10.879843</td>\n",
       "      <td>10.762563</td>\n",
       "      <td>10.671629</td>\n",
       "      <td>10.528853</td>\n",
       "      <td>10.370780</td>\n",
       "      <td>...</td>\n",
       "      <td>0.054570</td>\n",
       "      <td>0.249669</td>\n",
       "      <td>0.211801</td>\n",
       "      <td>0.182403</td>\n",
       "      <td>0.051033</td>\n",
       "      <td>0.051033</td>\n",
       "      <td>-0.035029</td>\n",
       "      <td>-0.069464</td>\n",
       "      <td>-0.087890</td>\n",
       "      <td>-0.088317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13277</th>\n",
       "      <td>318632</td>\n",
       "      <td>Fada N'gourma</td>\n",
       "      <td>2015</td>\n",
       "      <td>87</td>\n",
       "      <td>10.748116</td>\n",
       "      <td>10.528003</td>\n",
       "      <td>10.369930</td>\n",
       "      <td>10.135370</td>\n",
       "      <td>9.806476</td>\n",
       "      <td>9.706194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.027746</td>\n",
       "      <td>0.109486</td>\n",
       "      <td>0.059055</td>\n",
       "      <td>-0.041348</td>\n",
       "      <td>-0.080925</td>\n",
       "      <td>-0.080925</td>\n",
       "      <td>-0.223538</td>\n",
       "      <td>0.010742</td>\n",
       "      <td>-0.033203</td>\n",
       "      <td>-0.043557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4339</th>\n",
       "      <td>102549</td>\n",
       "      <td>Hounde</td>\n",
       "      <td>2015</td>\n",
       "      <td>116</td>\n",
       "      <td>9.152938</td>\n",
       "      <td>8.733959</td>\n",
       "      <td>8.262290</td>\n",
       "      <td>7.708185</td>\n",
       "      <td>7.514418</td>\n",
       "      <td>7.334248</td>\n",
       "      <td>...</td>\n",
       "      <td>0.049052</td>\n",
       "      <td>0.226975</td>\n",
       "      <td>0.100669</td>\n",
       "      <td>0.142427</td>\n",
       "      <td>-0.096613</td>\n",
       "      <td>-0.191706</td>\n",
       "      <td>-0.143885</td>\n",
       "      <td>0.079312</td>\n",
       "      <td>0.025346</td>\n",
       "      <td>-0.062499</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 250 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID        SettName  year  ADM_ID_x    AREA2015    AREA2014  \\\n",
       "9910    227047     Ouagadougou  2015       147  346.334596  338.023014   \n",
       "2177     51196  Bobo-Dioulasso  2015        37   88.690864   87.474722   \n",
       "8057    202965       Koudougou  2015       157   30.684014   30.157103   \n",
       "7            8         Banfora  2015        14   16.495685   16.022316   \n",
       "18503   456905      Ouahigouya  2015       219   13.275584   12.025448   \n",
       "5552    137698        Dedougou  2015        64   12.117232   11.673608   \n",
       "10721   247085       Tenkodogo  2015       296   11.164545   11.025168   \n",
       "12674   309529       Pouytenga  2015       249   10.984375   10.879843   \n",
       "13277   318632   Fada N'gourma  2015        87   10.748116   10.528003   \n",
       "4339    102549          Hounde  2015       116    9.152938    8.733959   \n",
       "\n",
       "         AREA2013    AREA2012    AREA2011    AREA2010  ...  NTLmax_pchD_2013  \\\n",
       "9910   326.382551  318.261337  310.990827  304.469890  ...          0.000000   \n",
       "2177    86.498238   84.906460   82.475025   80.240756  ...          0.000000   \n",
       "8057    29.583451   28.301020   27.843798   27.413772  ...          0.202622   \n",
       "7       15.331384   14.787476   14.615806   14.377846  ...          0.003737   \n",
       "18503   10.522054    9.225176    9.060304    8.580136  ...         -0.026280   \n",
       "5552    11.322618   10.958880   10.829702   10.616388  ...          0.011141   \n",
       "10721   10.938483   10.821203   10.260299    9.979847  ...          0.043578   \n",
       "12674   10.762563   10.671629   10.528853   10.370780  ...          0.054570   \n",
       "13277   10.369930   10.135370    9.806476    9.706194  ...          0.027746   \n",
       "4339     8.262290    7.708185    7.514418    7.334248  ...          0.049052   \n",
       "\n",
       "       NTLsum_pchV_2013  NTLavg_pchV_2013  NTLmax_pchV_2013  NTLsum_pchV_2014  \\\n",
       "9910           0.105390          0.046837         -0.036419          0.019402   \n",
       "2177           0.097519          0.070837         -0.015637         -0.033550   \n",
       "8057           0.540515          0.368111          1.384203         -0.128261   \n",
       "7              0.130527          0.139091          0.143354         -0.069417   \n",
       "18503          0.153354          0.098056          0.050841          0.062930   \n",
       "5552          -0.072531         -0.082294         -0.499691          0.475158   \n",
       "10721          0.110538          0.100796         -0.037817         -0.015631   \n",
       "12674          0.249669          0.211801          0.182403          0.051033   \n",
       "13277          0.109486          0.059055         -0.041348         -0.080925   \n",
       "4339           0.226975          0.100669          0.142427         -0.096613   \n",
       "\n",
       "       NTLavg_pchV_2014  NTLmax_pchV_2014  NTLsum_pchV_2015  NTLavg_pchV_2015  \\\n",
       "9910          -0.018029          0.093730         -0.006311         -0.057119   \n",
       "2177          -0.042858         -0.107115          0.074999          0.028763   \n",
       "8057          -0.118716         -0.243544         -0.056475         -0.059906   \n",
       "7             -0.096787         -0.154211          0.265583          0.265583   \n",
       "18503          0.034585          0.086015         -0.124337         -0.130137   \n",
       "5552           0.401400          0.176559          0.672705          0.371069   \n",
       "10721         -0.080180         -0.188068         -0.078570         -0.063213   \n",
       "12674          0.051033         -0.035029         -0.069464         -0.087890   \n",
       "13277         -0.080925         -0.223538          0.010742         -0.033203   \n",
       "4339          -0.191706         -0.143885          0.079312          0.025346   \n",
       "\n",
       "       NTLmax_pchV_2015  \n",
       "9910           0.156668  \n",
       "2177          -0.079833  \n",
       "8057          -0.317351  \n",
       "7             -0.061563  \n",
       "18503         -0.248345  \n",
       "5552           0.275277  \n",
       "10721         -0.304051  \n",
       "12674         -0.088317  \n",
       "13277         -0.043557  \n",
       "4339          -0.062499  \n",
       "\n",
       "[10 rows x 250 columns]"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(ResultsFolder, 'NTLChange.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "cef4598a",
   "metadata": {},
   "outputs": [],
   "source": [
    "AllStats.to_csv(os.path.join(ResultsFolder, 'AllStats.csv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d074141",
   "metadata": {},
   "source": [
    "### 11.3 Densities\n",
    "POPden = people per square kilometer\n",
    "<br>NTL...den = nighttime light luminosity per square kilometer\n",
    "<br>NTL...pop = nighttime light luminosity per capita"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b08da5d",
   "metadata": {},
   "source": [
    "#### Population Density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "501f3dfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(Population, how = 'outer', on='Sett_ID')\n",
    "Stats = Stats.merge(Areas, how='left', on='Sett_ID')\n",
    "\n",
    "for year in AllStudyYears:\n",
    "    RawVar = ''.join(['POPsum', str(year)])\n",
    "    DenomVar = ''.join(['AREA', str(year)])\n",
    "    NewVar = ''.join(['POPden', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "        Stats[NewVar] = Stats[RawVar] / Stats[DenomVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "20848ba4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Change in density\n",
    "for year in AllStudyYears:\n",
    "    RawVar = ''.join(['POPden', str(year)])\n",
    "    LagVar = ''.join(['POPden', str(year-1)])\n",
    "    NewVar = ''.join(['POPdenpch', str(year)])\n",
    "    if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "        Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "f3edb9a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Sett_ID', 'SettName', 'ADM_ID', 'POPden2000', 'POPden2001',\n",
       "       'POPden2002', 'POPden2003', 'POPden2004', 'POPden2005', 'POPden2006',\n",
       "       'POPden2007', 'POPden2008', 'POPden2009', 'POPden2010', 'POPden2011',\n",
       "       'POPden2012', 'POPden2013', 'POPden2014', 'POPden2015', 'POPdenpch2001',\n",
       "       'POPdenpch2002', 'POPdenpch2003', 'POPdenpch2004', 'POPdenpch2005',\n",
       "       'POPdenpch2006', 'POPdenpch2007', 'POPdenpch2008', 'POPdenpch2009',\n",
       "       'POPdenpch2010', 'POPdenpch2011', 'POPdenpch2012', 'POPdenpch2013',\n",
       "       'POPdenpch2014', 'POPdenpch2015'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, ~Stats.columns.str.contains('AREA|sum|ct|year')]\n",
    "Stats.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "08b74d6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>year</th>\n",
       "      <th>ADM_ID_x</th>\n",
       "      <th>AREA2015</th>\n",
       "      <th>AREA2014</th>\n",
       "      <th>AREA2013</th>\n",
       "      <th>AREA2012</th>\n",
       "      <th>AREA2011</th>\n",
       "      <th>AREA2010</th>\n",
       "      <th>...</th>\n",
       "      <th>POPdenpch2006</th>\n",
       "      <th>POPdenpch2007</th>\n",
       "      <th>POPdenpch2008</th>\n",
       "      <th>POPdenpch2009</th>\n",
       "      <th>POPdenpch2010</th>\n",
       "      <th>POPdenpch2011</th>\n",
       "      <th>POPdenpch2012</th>\n",
       "      <th>POPdenpch2013</th>\n",
       "      <th>POPdenpch2014</th>\n",
       "      <th>POPdenpch2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9910</th>\n",
       "      <td>227047</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>338.023014</td>\n",
       "      <td>326.382551</td>\n",
       "      <td>318.261337</td>\n",
       "      <td>310.990827</td>\n",
       "      <td>304.469890</td>\n",
       "      <td>...</td>\n",
       "      <td>0.046685</td>\n",
       "      <td>0.034082</td>\n",
       "      <td>0.052511</td>\n",
       "      <td>0.046055</td>\n",
       "      <td>0.016124</td>\n",
       "      <td>0.045698</td>\n",
       "      <td>0.038551</td>\n",
       "      <td>0.037351</td>\n",
       "      <td>0.025428</td>\n",
       "      <td>0.039638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2177</th>\n",
       "      <td>51196</td>\n",
       "      <td>Bobo-Dioulasso</td>\n",
       "      <td>2015</td>\n",
       "      <td>37</td>\n",
       "      <td>88.690864</td>\n",
       "      <td>87.474722</td>\n",
       "      <td>86.498238</td>\n",
       "      <td>84.906460</td>\n",
       "      <td>82.475025</td>\n",
       "      <td>80.240756</td>\n",
       "      <td>...</td>\n",
       "      <td>0.100682</td>\n",
       "      <td>-0.003081</td>\n",
       "      <td>0.002746</td>\n",
       "      <td>0.035042</td>\n",
       "      <td>0.001598</td>\n",
       "      <td>0.114985</td>\n",
       "      <td>-0.044462</td>\n",
       "      <td>0.057407</td>\n",
       "      <td>0.064245</td>\n",
       "      <td>-0.000379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8057</th>\n",
       "      <td>202965</td>\n",
       "      <td>Koudougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>157</td>\n",
       "      <td>30.684014</td>\n",
       "      <td>30.157103</td>\n",
       "      <td>29.583451</td>\n",
       "      <td>28.301020</td>\n",
       "      <td>27.843798</td>\n",
       "      <td>27.413772</td>\n",
       "      <td>...</td>\n",
       "      <td>0.041038</td>\n",
       "      <td>0.024490</td>\n",
       "      <td>-0.017747</td>\n",
       "      <td>-0.018505</td>\n",
       "      <td>0.024993</td>\n",
       "      <td>0.029423</td>\n",
       "      <td>-0.062943</td>\n",
       "      <td>0.016210</td>\n",
       "      <td>0.045434</td>\n",
       "      <td>-0.012297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Banfora</td>\n",
       "      <td>2015</td>\n",
       "      <td>14</td>\n",
       "      <td>16.495685</td>\n",
       "      <td>16.022316</td>\n",
       "      <td>15.331384</td>\n",
       "      <td>14.787476</td>\n",
       "      <td>14.615806</td>\n",
       "      <td>14.377846</td>\n",
       "      <td>...</td>\n",
       "      <td>0.015070</td>\n",
       "      <td>0.030276</td>\n",
       "      <td>-0.056810</td>\n",
       "      <td>0.025098</td>\n",
       "      <td>-0.021964</td>\n",
       "      <td>0.139976</td>\n",
       "      <td>-0.103608</td>\n",
       "      <td>0.056894</td>\n",
       "      <td>0.041396</td>\n",
       "      <td>-0.032538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18503</th>\n",
       "      <td>456905</td>\n",
       "      <td>Ouahigouya</td>\n",
       "      <td>2015</td>\n",
       "      <td>219</td>\n",
       "      <td>13.275584</td>\n",
       "      <td>12.025448</td>\n",
       "      <td>10.522054</td>\n",
       "      <td>9.225176</td>\n",
       "      <td>9.060304</td>\n",
       "      <td>8.580136</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.000358</td>\n",
       "      <td>-0.073643</td>\n",
       "      <td>0.034730</td>\n",
       "      <td>-0.065457</td>\n",
       "      <td>-0.048514</td>\n",
       "      <td>0.012432</td>\n",
       "      <td>-0.093699</td>\n",
       "      <td>-0.012318</td>\n",
       "      <td>-0.046107</td>\n",
       "      <td>-0.101715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5552</th>\n",
       "      <td>137698</td>\n",
       "      <td>Dedougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>64</td>\n",
       "      <td>12.117232</td>\n",
       "      <td>11.673608</td>\n",
       "      <td>11.322618</td>\n",
       "      <td>10.958880</td>\n",
       "      <td>10.829702</td>\n",
       "      <td>10.616388</td>\n",
       "      <td>...</td>\n",
       "      <td>0.164478</td>\n",
       "      <td>-0.114314</td>\n",
       "      <td>0.033625</td>\n",
       "      <td>-0.056484</td>\n",
       "      <td>-0.014446</td>\n",
       "      <td>0.131857</td>\n",
       "      <td>-0.040605</td>\n",
       "      <td>-0.038077</td>\n",
       "      <td>0.035645</td>\n",
       "      <td>0.113400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10721</th>\n",
       "      <td>247085</td>\n",
       "      <td>Tenkodogo</td>\n",
       "      <td>2015</td>\n",
       "      <td>296</td>\n",
       "      <td>11.164545</td>\n",
       "      <td>11.025168</td>\n",
       "      <td>10.938483</td>\n",
       "      <td>10.821203</td>\n",
       "      <td>10.260299</td>\n",
       "      <td>9.979847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.071305</td>\n",
       "      <td>0.019987</td>\n",
       "      <td>-0.021009</td>\n",
       "      <td>0.006384</td>\n",
       "      <td>-0.064241</td>\n",
       "      <td>-0.017585</td>\n",
       "      <td>-0.149854</td>\n",
       "      <td>0.019245</td>\n",
       "      <td>0.114017</td>\n",
       "      <td>0.051782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12674</th>\n",
       "      <td>309529</td>\n",
       "      <td>Pouytenga</td>\n",
       "      <td>2015</td>\n",
       "      <td>249</td>\n",
       "      <td>10.984375</td>\n",
       "      <td>10.879843</td>\n",
       "      <td>10.762563</td>\n",
       "      <td>10.671629</td>\n",
       "      <td>10.528853</td>\n",
       "      <td>10.370780</td>\n",
       "      <td>...</td>\n",
       "      <td>0.048217</td>\n",
       "      <td>0.045821</td>\n",
       "      <td>0.040806</td>\n",
       "      <td>-0.045779</td>\n",
       "      <td>-0.017349</td>\n",
       "      <td>0.061532</td>\n",
       "      <td>-0.038319</td>\n",
       "      <td>0.087220</td>\n",
       "      <td>0.111090</td>\n",
       "      <td>0.018788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13277</th>\n",
       "      <td>318632</td>\n",
       "      <td>Fada N'gourma</td>\n",
       "      <td>2015</td>\n",
       "      <td>87</td>\n",
       "      <td>10.748116</td>\n",
       "      <td>10.528003</td>\n",
       "      <td>10.369930</td>\n",
       "      <td>10.135370</td>\n",
       "      <td>9.806476</td>\n",
       "      <td>9.706194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.018000</td>\n",
       "      <td>-0.023178</td>\n",
       "      <td>0.078633</td>\n",
       "      <td>0.035921</td>\n",
       "      <td>-0.006863</td>\n",
       "      <td>0.070661</td>\n",
       "      <td>-0.060250</td>\n",
       "      <td>0.028201</td>\n",
       "      <td>0.019092</td>\n",
       "      <td>0.070414</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4339</th>\n",
       "      <td>102549</td>\n",
       "      <td>Hounde</td>\n",
       "      <td>2015</td>\n",
       "      <td>116</td>\n",
       "      <td>9.152938</td>\n",
       "      <td>8.733959</td>\n",
       "      <td>8.262290</td>\n",
       "      <td>7.708185</td>\n",
       "      <td>7.514418</td>\n",
       "      <td>7.334248</td>\n",
       "      <td>...</td>\n",
       "      <td>0.006776</td>\n",
       "      <td>-0.075117</td>\n",
       "      <td>-0.028986</td>\n",
       "      <td>-0.027155</td>\n",
       "      <td>0.003368</td>\n",
       "      <td>-0.022860</td>\n",
       "      <td>-0.111846</td>\n",
       "      <td>0.272436</td>\n",
       "      <td>0.071909</td>\n",
       "      <td>-0.014743</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 282 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID        SettName  year  ADM_ID_x    AREA2015    AREA2014  \\\n",
       "9910    227047     Ouagadougou  2015       147  346.334596  338.023014   \n",
       "2177     51196  Bobo-Dioulasso  2015        37   88.690864   87.474722   \n",
       "8057    202965       Koudougou  2015       157   30.684014   30.157103   \n",
       "7            8         Banfora  2015        14   16.495685   16.022316   \n",
       "18503   456905      Ouahigouya  2015       219   13.275584   12.025448   \n",
       "5552    137698        Dedougou  2015        64   12.117232   11.673608   \n",
       "10721   247085       Tenkodogo  2015       296   11.164545   11.025168   \n",
       "12674   309529       Pouytenga  2015       249   10.984375   10.879843   \n",
       "13277   318632   Fada N'gourma  2015        87   10.748116   10.528003   \n",
       "4339    102549          Hounde  2015       116    9.152938    8.733959   \n",
       "\n",
       "         AREA2013    AREA2012    AREA2011    AREA2010  ...  POPdenpch2006  \\\n",
       "9910   326.382551  318.261337  310.990827  304.469890  ...       0.046685   \n",
       "2177    86.498238   84.906460   82.475025   80.240756  ...       0.100682   \n",
       "8057    29.583451   28.301020   27.843798   27.413772  ...       0.041038   \n",
       "7       15.331384   14.787476   14.615806   14.377846  ...       0.015070   \n",
       "18503   10.522054    9.225176    9.060304    8.580136  ...      -0.000358   \n",
       "5552    11.322618   10.958880   10.829702   10.616388  ...       0.164478   \n",
       "10721   10.938483   10.821203   10.260299    9.979847  ...       0.071305   \n",
       "12674   10.762563   10.671629   10.528853   10.370780  ...       0.048217   \n",
       "13277   10.369930   10.135370    9.806476    9.706194  ...       0.018000   \n",
       "4339     8.262290    7.708185    7.514418    7.334248  ...       0.006776   \n",
       "\n",
       "       POPdenpch2007  POPdenpch2008  POPdenpch2009  POPdenpch2010  \\\n",
       "9910        0.034082       0.052511       0.046055       0.016124   \n",
       "2177       -0.003081       0.002746       0.035042       0.001598   \n",
       "8057        0.024490      -0.017747      -0.018505       0.024993   \n",
       "7           0.030276      -0.056810       0.025098      -0.021964   \n",
       "18503      -0.073643       0.034730      -0.065457      -0.048514   \n",
       "5552       -0.114314       0.033625      -0.056484      -0.014446   \n",
       "10721       0.019987      -0.021009       0.006384      -0.064241   \n",
       "12674       0.045821       0.040806      -0.045779      -0.017349   \n",
       "13277      -0.023178       0.078633       0.035921      -0.006863   \n",
       "4339       -0.075117      -0.028986      -0.027155       0.003368   \n",
       "\n",
       "       POPdenpch2011  POPdenpch2012  POPdenpch2013  POPdenpch2014  \\\n",
       "9910        0.045698       0.038551       0.037351       0.025428   \n",
       "2177        0.114985      -0.044462       0.057407       0.064245   \n",
       "8057        0.029423      -0.062943       0.016210       0.045434   \n",
       "7           0.139976      -0.103608       0.056894       0.041396   \n",
       "18503       0.012432      -0.093699      -0.012318      -0.046107   \n",
       "5552        0.131857      -0.040605      -0.038077       0.035645   \n",
       "10721      -0.017585      -0.149854       0.019245       0.114017   \n",
       "12674       0.061532      -0.038319       0.087220       0.111090   \n",
       "13277       0.070661      -0.060250       0.028201       0.019092   \n",
       "4339       -0.022860      -0.111846       0.272436       0.071909   \n",
       "\n",
       "       POPdenpch2015  \n",
       "9910        0.039638  \n",
       "2177       -0.000379  \n",
       "8057       -0.012297  \n",
       "7          -0.032538  \n",
       "18503      -0.101715  \n",
       "5552        0.113400  \n",
       "10721       0.051782  \n",
       "12674       0.018788  \n",
       "13277       0.070414  \n",
       "4339       -0.014743  \n",
       "\n",
       "[10 rows x 282 columns]"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(ResultsFolder, 'PopDensity.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09b9e3a3",
   "metadata": {},
   "source": [
    "#### Nighttime Lights Density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "8ef877b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(NTL, how = 'outer', on='Sett_ID')\n",
    "Stats = Stats.merge(Areas, how='left', on='Sett_ID')\n",
    "Sensors = ['D_', 'V_']\n",
    "Methods = ['sum', 'avg', 'max']\n",
    "\n",
    "for year in AllStudyYears:\n",
    "    for Sensor in Sensors:\n",
    "        for agg in Methods:\n",
    "            RawVar = ''.join(['NTL', agg, Sensor, str(year)])\n",
    "            DenomVar = ''.join(['AREA', str(year)])\n",
    "            NewVar = ''.join(['NTL', agg, '_den', Sensor, str(year)])\n",
    "            if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "                Stats[NewVar] = Stats[RawVar] / Stats[DenomVar]\n",
    "            else:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "606d6807",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\1631515177.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n"
     ]
    }
   ],
   "source": [
    "# Change in density\n",
    "for year in AllStudyYears:\n",
    "    for Sensor in Sensors:\n",
    "        for agg in Methods:\n",
    "            RawVar = ''.join(['NTL', agg, '_den', Sensor, str(year)])\n",
    "            LagVar = ''.join(['NTL', agg, '_den', Sensor, str(year-1)])\n",
    "            NewVar = ''.join(['NTL', agg, '_denpch', Sensor, str(year)])\n",
    "            if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "                Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "            else:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "d18b4f08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sett_ID',\n",
       " 'SettName',\n",
       " 'NTLctD_1999',\n",
       " 'NTLsumD_1999',\n",
       " 'NTLavgD_1999',\n",
       " 'NTLmaxD_1999',\n",
       " 'NTLminD_1999',\n",
       " 'NTLcfc_D_1999',\n",
       " 'NTLctD_2000',\n",
       " 'NTLsumD_2000',\n",
       " 'NTLavgD_2000',\n",
       " 'NTLmaxD_2000',\n",
       " 'NTLminD_2000',\n",
       " 'NTLcfc_D_2000',\n",
       " 'NTLctD_2001',\n",
       " 'NTLsumD_2001',\n",
       " 'NTLavgD_2001',\n",
       " 'NTLmaxD_2001',\n",
       " 'NTLminD_2001',\n",
       " 'NTLcfc_D_2001',\n",
       " 'NTLctD_2002',\n",
       " 'NTLsumD_2002',\n",
       " 'NTLavgD_2002',\n",
       " 'NTLmaxD_2002',\n",
       " 'NTLminD_2002',\n",
       " 'NTLcfc_D_2002',\n",
       " 'NTLctD_2003',\n",
       " 'NTLsumD_2003',\n",
       " 'NTLavgD_2003',\n",
       " 'NTLmaxD_2003',\n",
       " 'NTLminD_2003',\n",
       " 'NTLcfc_D_2003',\n",
       " 'NTLctD_2004',\n",
       " 'NTLsumD_2004',\n",
       " 'NTLavgD_2004',\n",
       " 'NTLmaxD_2004',\n",
       " 'NTLminD_2004',\n",
       " 'NTLcfc_D_2004',\n",
       " 'NTLctD_2005',\n",
       " 'NTLsumD_2005',\n",
       " 'NTLavgD_2005',\n",
       " 'NTLmaxD_2005',\n",
       " 'NTLminD_2005',\n",
       " 'NTLcfc_D_2005',\n",
       " 'NTLctD_2006',\n",
       " 'NTLsumD_2006',\n",
       " 'NTLavgD_2006',\n",
       " 'NTLmaxD_2006',\n",
       " 'NTLminD_2006',\n",
       " 'NTLcfc_D_2006',\n",
       " 'NTLctD_2007',\n",
       " 'NTLsumD_2007',\n",
       " 'NTLavgD_2007',\n",
       " 'NTLmaxD_2007',\n",
       " 'NTLminD_2007',\n",
       " 'NTLcfc_D_2007',\n",
       " 'NTLctD_2008',\n",
       " 'NTLsumD_2008',\n",
       " 'NTLavgD_2008',\n",
       " 'NTLmaxD_2008',\n",
       " 'NTLminD_2008',\n",
       " 'NTLcfc_D_2008',\n",
       " 'NTLctD_2009',\n",
       " 'NTLsumD_2009',\n",
       " 'NTLavgD_2009',\n",
       " 'NTLmaxD_2009',\n",
       " 'NTLminD_2009',\n",
       " 'NTLcfc_D_2009',\n",
       " 'NTLctD_2010',\n",
       " 'NTLsumD_2010',\n",
       " 'NTLavgD_2010',\n",
       " 'NTLmaxD_2010',\n",
       " 'NTLminD_2010',\n",
       " 'NTLcfc_D_2010',\n",
       " 'NTLctD_2011',\n",
       " 'NTLsumD_2011',\n",
       " 'NTLavgD_2011',\n",
       " 'NTLmaxD_2011',\n",
       " 'NTLminD_2011',\n",
       " 'NTLcfc_D_2011',\n",
       " 'NTLctD_2012',\n",
       " 'NTLsumD_2012',\n",
       " 'NTLavgD_2012',\n",
       " 'NTLmaxD_2012',\n",
       " 'NTLminD_2012',\n",
       " 'NTLcfc_D_2012',\n",
       " 'NTLctD_2013',\n",
       " 'NTLsumD_2013',\n",
       " 'NTLavgD_2013',\n",
       " 'NTLmaxD_2013',\n",
       " 'NTLminD_2013',\n",
       " 'NTLcfc_D_2013',\n",
       " 'NTLctV_2012',\n",
       " 'NTLsumV_2012',\n",
       " 'NTLavgV_2012',\n",
       " 'NTLmaxV_2012',\n",
       " 'NTLminV_2012',\n",
       " 'NTLcfc_V_2012',\n",
       " 'NTLctV_2013',\n",
       " 'NTLsumV_2013',\n",
       " 'NTLavgV_2013',\n",
       " 'NTLmaxV_2013',\n",
       " 'NTLminV_2013',\n",
       " 'NTLcfc_V_2013',\n",
       " 'NTLctV_2014',\n",
       " 'NTLsumV_2014',\n",
       " 'NTLavgV_2014',\n",
       " 'NTLmaxV_2014',\n",
       " 'NTLminV_2014',\n",
       " 'NTLcfc_V_2014',\n",
       " 'NTLctV_2015',\n",
       " 'NTLsumV_2015',\n",
       " 'NTLavgV_2015',\n",
       " 'NTLmaxV_2015',\n",
       " 'NTLminV_2015',\n",
       " 'NTLcfc_V_2015',\n",
       " 'year',\n",
       " 'ADM_ID',\n",
       " 'AREA2015',\n",
       " 'AREA2014',\n",
       " 'AREA2013',\n",
       " 'AREA2012',\n",
       " 'AREA2011',\n",
       " 'AREA2010',\n",
       " 'AREA2009',\n",
       " 'AREA2008',\n",
       " 'AREA2007',\n",
       " 'AREA2006',\n",
       " 'AREA2005',\n",
       " 'AREA2004',\n",
       " 'AREA2003',\n",
       " 'AREA2002',\n",
       " 'AREA2001',\n",
       " 'AREA2000',\n",
       " 'AREA1999',\n",
       " 'NTLsum_denD_1999',\n",
       " 'NTLavg_denD_1999',\n",
       " 'NTLmax_denD_1999',\n",
       " 'NTLsum_denD_2000',\n",
       " 'NTLavg_denD_2000',\n",
       " 'NTLmax_denD_2000',\n",
       " 'NTLsum_denD_2001',\n",
       " 'NTLavg_denD_2001',\n",
       " 'NTLmax_denD_2001',\n",
       " 'NTLsum_denD_2002',\n",
       " 'NTLavg_denD_2002',\n",
       " 'NTLmax_denD_2002',\n",
       " 'NTLsum_denD_2003',\n",
       " 'NTLavg_denD_2003',\n",
       " 'NTLmax_denD_2003',\n",
       " 'NTLsum_denD_2004',\n",
       " 'NTLavg_denD_2004',\n",
       " 'NTLmax_denD_2004',\n",
       " 'NTLsum_denD_2005',\n",
       " 'NTLavg_denD_2005',\n",
       " 'NTLmax_denD_2005',\n",
       " 'NTLsum_denD_2006',\n",
       " 'NTLavg_denD_2006',\n",
       " 'NTLmax_denD_2006',\n",
       " 'NTLsum_denD_2007',\n",
       " 'NTLavg_denD_2007',\n",
       " 'NTLmax_denD_2007',\n",
       " 'NTLsum_denD_2008',\n",
       " 'NTLavg_denD_2008',\n",
       " 'NTLmax_denD_2008',\n",
       " 'NTLsum_denD_2009',\n",
       " 'NTLavg_denD_2009',\n",
       " 'NTLmax_denD_2009',\n",
       " 'NTLsum_denD_2010',\n",
       " 'NTLavg_denD_2010',\n",
       " 'NTLmax_denD_2010',\n",
       " 'NTLsum_denD_2011',\n",
       " 'NTLavg_denD_2011',\n",
       " 'NTLmax_denD_2011',\n",
       " 'NTLsum_denD_2012',\n",
       " 'NTLavg_denD_2012',\n",
       " 'NTLmax_denD_2012',\n",
       " 'NTLsum_denV_2012',\n",
       " 'NTLavg_denV_2012',\n",
       " 'NTLmax_denV_2012',\n",
       " 'NTLsum_denD_2013',\n",
       " 'NTLavg_denD_2013',\n",
       " 'NTLmax_denD_2013',\n",
       " 'NTLsum_denV_2013',\n",
       " 'NTLavg_denV_2013',\n",
       " 'NTLmax_denV_2013',\n",
       " 'NTLsum_denV_2014',\n",
       " 'NTLavg_denV_2014',\n",
       " 'NTLmax_denV_2014',\n",
       " 'NTLsum_denV_2015',\n",
       " 'NTLavg_denV_2015',\n",
       " 'NTLmax_denV_2015',\n",
       " 'NTLsum_denpchD_2000',\n",
       " 'NTLavg_denpchD_2000',\n",
       " 'NTLmax_denpchD_2000',\n",
       " 'NTLsum_denpchD_2001',\n",
       " 'NTLavg_denpchD_2001',\n",
       " 'NTLmax_denpchD_2001',\n",
       " 'NTLsum_denpchD_2002',\n",
       " 'NTLavg_denpchD_2002',\n",
       " 'NTLmax_denpchD_2002',\n",
       " 'NTLsum_denpchD_2003',\n",
       " 'NTLavg_denpchD_2003',\n",
       " 'NTLmax_denpchD_2003',\n",
       " 'NTLsum_denpchD_2004',\n",
       " 'NTLavg_denpchD_2004',\n",
       " 'NTLmax_denpchD_2004',\n",
       " 'NTLsum_denpchD_2005',\n",
       " 'NTLavg_denpchD_2005',\n",
       " 'NTLmax_denpchD_2005',\n",
       " 'NTLsum_denpchD_2006',\n",
       " 'NTLavg_denpchD_2006',\n",
       " 'NTLmax_denpchD_2006',\n",
       " 'NTLsum_denpchD_2007',\n",
       " 'NTLavg_denpchD_2007',\n",
       " 'NTLmax_denpchD_2007',\n",
       " 'NTLsum_denpchD_2008',\n",
       " 'NTLavg_denpchD_2008',\n",
       " 'NTLmax_denpchD_2008',\n",
       " 'NTLsum_denpchD_2009',\n",
       " 'NTLavg_denpchD_2009',\n",
       " 'NTLmax_denpchD_2009',\n",
       " 'NTLsum_denpchD_2010',\n",
       " 'NTLavg_denpchD_2010',\n",
       " 'NTLmax_denpchD_2010',\n",
       " 'NTLsum_denpchD_2011',\n",
       " 'NTLavg_denpchD_2011',\n",
       " 'NTLmax_denpchD_2011',\n",
       " 'NTLsum_denpchD_2012',\n",
       " 'NTLavg_denpchD_2012',\n",
       " 'NTLmax_denpchD_2012',\n",
       " 'NTLsum_denpchD_2013',\n",
       " 'NTLavg_denpchD_2013',\n",
       " 'NTLmax_denpchD_2013',\n",
       " 'NTLsum_denpchV_2013',\n",
       " 'NTLavg_denpchV_2013',\n",
       " 'NTLmax_denpchV_2013',\n",
       " 'NTLsum_denpchV_2014',\n",
       " 'NTLavg_denpchV_2014',\n",
       " 'NTLmax_denpchV_2014',\n",
       " 'NTLsum_denpchV_2015',\n",
       " 'NTLavg_denpchV_2015',\n",
       " 'NTLmax_denpchV_2015']"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "f9ca39bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sett_ID',\n",
       " 'SettName',\n",
       " 'NTLsum_denD_1999',\n",
       " 'NTLavg_denD_1999',\n",
       " 'NTLmax_denD_1999',\n",
       " 'NTLsum_denD_2000',\n",
       " 'NTLavg_denD_2000',\n",
       " 'NTLmax_denD_2000',\n",
       " 'NTLsum_denD_2001',\n",
       " 'NTLavg_denD_2001',\n",
       " 'NTLmax_denD_2001',\n",
       " 'NTLsum_denD_2002',\n",
       " 'NTLavg_denD_2002',\n",
       " 'NTLmax_denD_2002',\n",
       " 'NTLsum_denD_2003',\n",
       " 'NTLavg_denD_2003',\n",
       " 'NTLmax_denD_2003',\n",
       " 'NTLsum_denD_2004',\n",
       " 'NTLavg_denD_2004',\n",
       " 'NTLmax_denD_2004',\n",
       " 'NTLsum_denD_2005',\n",
       " 'NTLavg_denD_2005',\n",
       " 'NTLmax_denD_2005',\n",
       " 'NTLsum_denD_2006',\n",
       " 'NTLavg_denD_2006',\n",
       " 'NTLmax_denD_2006',\n",
       " 'NTLsum_denD_2007',\n",
       " 'NTLavg_denD_2007',\n",
       " 'NTLmax_denD_2007',\n",
       " 'NTLsum_denD_2008',\n",
       " 'NTLavg_denD_2008',\n",
       " 'NTLmax_denD_2008',\n",
       " 'NTLsum_denD_2009',\n",
       " 'NTLavg_denD_2009',\n",
       " 'NTLmax_denD_2009',\n",
       " 'NTLsum_denD_2010',\n",
       " 'NTLavg_denD_2010',\n",
       " 'NTLmax_denD_2010',\n",
       " 'NTLsum_denD_2011',\n",
       " 'NTLavg_denD_2011',\n",
       " 'NTLmax_denD_2011',\n",
       " 'NTLsum_denD_2012',\n",
       " 'NTLavg_denD_2012',\n",
       " 'NTLmax_denD_2012',\n",
       " 'NTLsum_denV_2012',\n",
       " 'NTLavg_denV_2012',\n",
       " 'NTLmax_denV_2012',\n",
       " 'NTLsum_denD_2013',\n",
       " 'NTLavg_denD_2013',\n",
       " 'NTLmax_denD_2013',\n",
       " 'NTLsum_denV_2013',\n",
       " 'NTLavg_denV_2013',\n",
       " 'NTLmax_denV_2013',\n",
       " 'NTLsum_denV_2014',\n",
       " 'NTLavg_denV_2014',\n",
       " 'NTLmax_denV_2014',\n",
       " 'NTLsum_denV_2015',\n",
       " 'NTLavg_denV_2015',\n",
       " 'NTLmax_denV_2015',\n",
       " 'NTLsum_denpchD_2000',\n",
       " 'NTLavg_denpchD_2000',\n",
       " 'NTLmax_denpchD_2000',\n",
       " 'NTLsum_denpchD_2001',\n",
       " 'NTLavg_denpchD_2001',\n",
       " 'NTLmax_denpchD_2001',\n",
       " 'NTLsum_denpchD_2002',\n",
       " 'NTLavg_denpchD_2002',\n",
       " 'NTLmax_denpchD_2002',\n",
       " 'NTLsum_denpchD_2003',\n",
       " 'NTLavg_denpchD_2003',\n",
       " 'NTLmax_denpchD_2003',\n",
       " 'NTLsum_denpchD_2004',\n",
       " 'NTLavg_denpchD_2004',\n",
       " 'NTLmax_denpchD_2004',\n",
       " 'NTLsum_denpchD_2005',\n",
       " 'NTLavg_denpchD_2005',\n",
       " 'NTLmax_denpchD_2005',\n",
       " 'NTLsum_denpchD_2006',\n",
       " 'NTLavg_denpchD_2006',\n",
       " 'NTLmax_denpchD_2006',\n",
       " 'NTLsum_denpchD_2007',\n",
       " 'NTLavg_denpchD_2007',\n",
       " 'NTLmax_denpchD_2007',\n",
       " 'NTLsum_denpchD_2008',\n",
       " 'NTLavg_denpchD_2008',\n",
       " 'NTLmax_denpchD_2008',\n",
       " 'NTLsum_denpchD_2009',\n",
       " 'NTLavg_denpchD_2009',\n",
       " 'NTLmax_denpchD_2009',\n",
       " 'NTLsum_denpchD_2010',\n",
       " 'NTLavg_denpchD_2010',\n",
       " 'NTLmax_denpchD_2010',\n",
       " 'NTLsum_denpchD_2011',\n",
       " 'NTLavg_denpchD_2011',\n",
       " 'NTLmax_denpchD_2011',\n",
       " 'NTLsum_denpchD_2012',\n",
       " 'NTLavg_denpchD_2012',\n",
       " 'NTLmax_denpchD_2012',\n",
       " 'NTLsum_denpchD_2013',\n",
       " 'NTLavg_denpchD_2013',\n",
       " 'NTLmax_denpchD_2013',\n",
       " 'NTLsum_denpchV_2013',\n",
       " 'NTLavg_denpchV_2013',\n",
       " 'NTLmax_denpchV_2013',\n",
       " 'NTLsum_denpchV_2014',\n",
       " 'NTLavg_denpchV_2014',\n",
       " 'NTLmax_denpchV_2014',\n",
       " 'NTLsum_denpchV_2015',\n",
       " 'NTLavg_denpchV_2015',\n",
       " 'NTLmax_denpchV_2015']"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|den')]\n",
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "720e5a3d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>year</th>\n",
       "      <th>ADM_ID_x</th>\n",
       "      <th>AREA2015</th>\n",
       "      <th>AREA2014</th>\n",
       "      <th>AREA2013</th>\n",
       "      <th>AREA2012</th>\n",
       "      <th>AREA2011</th>\n",
       "      <th>AREA2010</th>\n",
       "      <th>...</th>\n",
       "      <th>NTLmax_denpchD_2013</th>\n",
       "      <th>NTLsum_denpchV_2013</th>\n",
       "      <th>NTLavg_denpchV_2013</th>\n",
       "      <th>NTLmax_denpchV_2013</th>\n",
       "      <th>NTLsum_denpchV_2014</th>\n",
       "      <th>NTLavg_denpchV_2014</th>\n",
       "      <th>NTLmax_denpchV_2014</th>\n",
       "      <th>NTLsum_denpchV_2015</th>\n",
       "      <th>NTLavg_denpchV_2015</th>\n",
       "      <th>NTLmax_denpchV_2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9910</th>\n",
       "      <td>227047</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>338.023014</td>\n",
       "      <td>326.382551</td>\n",
       "      <td>318.261337</td>\n",
       "      <td>310.990827</td>\n",
       "      <td>304.469890</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.024883</td>\n",
       "      <td>0.077885</td>\n",
       "      <td>0.020789</td>\n",
       "      <td>-0.060396</td>\n",
       "      <td>-0.015703</td>\n",
       "      <td>-0.051845</td>\n",
       "      <td>0.056066</td>\n",
       "      <td>-0.030158</td>\n",
       "      <td>-0.079747</td>\n",
       "      <td>0.128910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2177</th>\n",
       "      <td>51196</td>\n",
       "      <td>Bobo-Dioulasso</td>\n",
       "      <td>2015</td>\n",
       "      <td>37</td>\n",
       "      <td>88.690864</td>\n",
       "      <td>87.474722</td>\n",
       "      <td>86.498238</td>\n",
       "      <td>84.906460</td>\n",
       "      <td>82.475025</td>\n",
       "      <td>80.240756</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.018402</td>\n",
       "      <td>0.077322</td>\n",
       "      <td>0.051131</td>\n",
       "      <td>-0.033751</td>\n",
       "      <td>-0.044339</td>\n",
       "      <td>-0.053543</td>\n",
       "      <td>-0.117083</td>\n",
       "      <td>0.060259</td>\n",
       "      <td>0.014656</td>\n",
       "      <td>-0.092450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8057</th>\n",
       "      <td>202965</td>\n",
       "      <td>Koudougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>157</td>\n",
       "      <td>30.684014</td>\n",
       "      <td>30.157103</td>\n",
       "      <td>29.583451</td>\n",
       "      <td>28.301020</td>\n",
       "      <td>27.843798</td>\n",
       "      <td>27.413772</td>\n",
       "      <td>...</td>\n",
       "      <td>0.150489</td>\n",
       "      <td>0.473734</td>\n",
       "      <td>0.308804</td>\n",
       "      <td>1.280848</td>\n",
       "      <td>-0.144843</td>\n",
       "      <td>-0.135480</td>\n",
       "      <td>-0.257933</td>\n",
       "      <td>-0.072677</td>\n",
       "      <td>-0.076049</td>\n",
       "      <td>-0.329074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Banfora</td>\n",
       "      <td>2015</td>\n",
       "      <td>14</td>\n",
       "      <td>16.495685</td>\n",
       "      <td>16.022316</td>\n",
       "      <td>15.331384</td>\n",
       "      <td>14.787476</td>\n",
       "      <td>14.615806</td>\n",
       "      <td>14.377846</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.031872</td>\n",
       "      <td>0.090419</td>\n",
       "      <td>0.098680</td>\n",
       "      <td>0.102791</td>\n",
       "      <td>-0.109547</td>\n",
       "      <td>-0.135737</td>\n",
       "      <td>-0.190685</td>\n",
       "      <td>0.229265</td>\n",
       "      <td>0.229265</td>\n",
       "      <td>-0.088493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18503</th>\n",
       "      <td>456905</td>\n",
       "      <td>Ouahigouya</td>\n",
       "      <td>2015</td>\n",
       "      <td>219</td>\n",
       "      <td>13.275584</td>\n",
       "      <td>12.025448</td>\n",
       "      <td>10.522054</td>\n",
       "      <td>9.225176</td>\n",
       "      <td>9.060304</td>\n",
       "      <td>8.580136</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.146294</td>\n",
       "      <td>0.011199</td>\n",
       "      <td>-0.037283</td>\n",
       "      <td>-0.078679</td>\n",
       "      <td>-0.069955</td>\n",
       "      <td>-0.094756</td>\n",
       "      <td>-0.049756</td>\n",
       "      <td>-0.206797</td>\n",
       "      <td>-0.212050</td>\n",
       "      <td>-0.319127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5552</th>\n",
       "      <td>137698</td>\n",
       "      <td>Dedougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>64</td>\n",
       "      <td>12.117232</td>\n",
       "      <td>11.673608</td>\n",
       "      <td>11.322618</td>\n",
       "      <td>10.958880</td>\n",
       "      <td>10.829702</td>\n",
       "      <td>10.616388</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.021342</td>\n",
       "      <td>-0.102326</td>\n",
       "      <td>-0.111775</td>\n",
       "      <td>-0.515764</td>\n",
       "      <td>0.430805</td>\n",
       "      <td>0.359264</td>\n",
       "      <td>0.141184</td>\n",
       "      <td>0.611465</td>\n",
       "      <td>0.320873</td>\n",
       "      <td>0.228587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10721</th>\n",
       "      <td>247085</td>\n",
       "      <td>Tenkodogo</td>\n",
       "      <td>2015</td>\n",
       "      <td>296</td>\n",
       "      <td>11.164545</td>\n",
       "      <td>11.025168</td>\n",
       "      <td>10.938483</td>\n",
       "      <td>10.821203</td>\n",
       "      <td>10.260299</td>\n",
       "      <td>9.979847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.032389</td>\n",
       "      <td>0.098631</td>\n",
       "      <td>0.088994</td>\n",
       "      <td>-0.048133</td>\n",
       "      <td>-0.023371</td>\n",
       "      <td>-0.087412</td>\n",
       "      <td>-0.194452</td>\n",
       "      <td>-0.090073</td>\n",
       "      <td>-0.074907</td>\n",
       "      <td>-0.312739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12674</th>\n",
       "      <td>309529</td>\n",
       "      <td>Pouytenga</td>\n",
       "      <td>2015</td>\n",
       "      <td>249</td>\n",
       "      <td>10.984375</td>\n",
       "      <td>10.879843</td>\n",
       "      <td>10.762563</td>\n",
       "      <td>10.671629</td>\n",
       "      <td>10.528853</td>\n",
       "      <td>10.370780</td>\n",
       "      <td>...</td>\n",
       "      <td>0.045660</td>\n",
       "      <td>0.239111</td>\n",
       "      <td>0.201562</td>\n",
       "      <td>0.172412</td>\n",
       "      <td>0.039703</td>\n",
       "      <td>0.039703</td>\n",
       "      <td>-0.045431</td>\n",
       "      <td>-0.078319</td>\n",
       "      <td>-0.096570</td>\n",
       "      <td>-0.096993</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13277</th>\n",
       "      <td>318632</td>\n",
       "      <td>Fada N'gourma</td>\n",
       "      <td>2015</td>\n",
       "      <td>87</td>\n",
       "      <td>10.748116</td>\n",
       "      <td>10.528003</td>\n",
       "      <td>10.369930</td>\n",
       "      <td>10.135370</td>\n",
       "      <td>9.806476</td>\n",
       "      <td>9.706194</td>\n",
       "      <td>...</td>\n",
       "      <td>0.004499</td>\n",
       "      <td>0.084391</td>\n",
       "      <td>0.035100</td>\n",
       "      <td>-0.063032</td>\n",
       "      <td>-0.094725</td>\n",
       "      <td>-0.094725</td>\n",
       "      <td>-0.235196</td>\n",
       "      <td>-0.009957</td>\n",
       "      <td>-0.053002</td>\n",
       "      <td>-0.063145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4339</th>\n",
       "      <td>102549</td>\n",
       "      <td>Hounde</td>\n",
       "      <td>2015</td>\n",
       "      <td>116</td>\n",
       "      <td>9.152938</td>\n",
       "      <td>8.733959</td>\n",
       "      <td>8.262290</td>\n",
       "      <td>7.708185</td>\n",
       "      <td>7.514418</td>\n",
       "      <td>7.334248</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.021302</td>\n",
       "      <td>0.144688</td>\n",
       "      <td>0.026853</td>\n",
       "      <td>0.065811</td>\n",
       "      <td>-0.145399</td>\n",
       "      <td>-0.235357</td>\n",
       "      <td>-0.190119</td>\n",
       "      <td>0.029906</td>\n",
       "      <td>-0.021590</td>\n",
       "      <td>-0.105414</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 390 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID        SettName  year  ADM_ID_x    AREA2015    AREA2014  \\\n",
       "9910    227047     Ouagadougou  2015       147  346.334596  338.023014   \n",
       "2177     51196  Bobo-Dioulasso  2015        37   88.690864   87.474722   \n",
       "8057    202965       Koudougou  2015       157   30.684014   30.157103   \n",
       "7            8         Banfora  2015        14   16.495685   16.022316   \n",
       "18503   456905      Ouahigouya  2015       219   13.275584   12.025448   \n",
       "5552    137698        Dedougou  2015        64   12.117232   11.673608   \n",
       "10721   247085       Tenkodogo  2015       296   11.164545   11.025168   \n",
       "12674   309529       Pouytenga  2015       249   10.984375   10.879843   \n",
       "13277   318632   Fada N'gourma  2015        87   10.748116   10.528003   \n",
       "4339    102549          Hounde  2015       116    9.152938    8.733959   \n",
       "\n",
       "         AREA2013    AREA2012    AREA2011    AREA2010  ...  \\\n",
       "9910   326.382551  318.261337  310.990827  304.469890  ...   \n",
       "2177    86.498238   84.906460   82.475025   80.240756  ...   \n",
       "8057    29.583451   28.301020   27.843798   27.413772  ...   \n",
       "7       15.331384   14.787476   14.615806   14.377846  ...   \n",
       "18503   10.522054    9.225176    9.060304    8.580136  ...   \n",
       "5552    11.322618   10.958880   10.829702   10.616388  ...   \n",
       "10721   10.938483   10.821203   10.260299    9.979847  ...   \n",
       "12674   10.762563   10.671629   10.528853   10.370780  ...   \n",
       "13277   10.369930   10.135370    9.806476    9.706194  ...   \n",
       "4339     8.262290    7.708185    7.514418    7.334248  ...   \n",
       "\n",
       "       NTLmax_denpchD_2013  NTLsum_denpchV_2013  NTLavg_denpchV_2013  \\\n",
       "9910             -0.024883             0.077885             0.020789   \n",
       "2177             -0.018402             0.077322             0.051131   \n",
       "8057              0.150489             0.473734             0.308804   \n",
       "7                -0.031872             0.090419             0.098680   \n",
       "18503            -0.146294             0.011199            -0.037283   \n",
       "5552             -0.021342            -0.102326            -0.111775   \n",
       "10721             0.032389             0.098631             0.088994   \n",
       "12674             0.045660             0.239111             0.201562   \n",
       "13277             0.004499             0.084391             0.035100   \n",
       "4339             -0.021302             0.144688             0.026853   \n",
       "\n",
       "       NTLmax_denpchV_2013  NTLsum_denpchV_2014  NTLavg_denpchV_2014  \\\n",
       "9910             -0.060396            -0.015703            -0.051845   \n",
       "2177             -0.033751            -0.044339            -0.053543   \n",
       "8057              1.280848            -0.144843            -0.135480   \n",
       "7                 0.102791            -0.109547            -0.135737   \n",
       "18503            -0.078679            -0.069955            -0.094756   \n",
       "5552             -0.515764             0.430805             0.359264   \n",
       "10721            -0.048133            -0.023371            -0.087412   \n",
       "12674             0.172412             0.039703             0.039703   \n",
       "13277            -0.063032            -0.094725            -0.094725   \n",
       "4339              0.065811            -0.145399            -0.235357   \n",
       "\n",
       "       NTLmax_denpchV_2014  NTLsum_denpchV_2015  NTLavg_denpchV_2015  \\\n",
       "9910              0.056066            -0.030158            -0.079747   \n",
       "2177             -0.117083             0.060259             0.014656   \n",
       "8057             -0.257933            -0.072677            -0.076049   \n",
       "7                -0.190685             0.229265             0.229265   \n",
       "18503            -0.049756            -0.206797            -0.212050   \n",
       "5552              0.141184             0.611465             0.320873   \n",
       "10721            -0.194452            -0.090073            -0.074907   \n",
       "12674            -0.045431            -0.078319            -0.096570   \n",
       "13277            -0.235196            -0.009957            -0.053002   \n",
       "4339             -0.190119             0.029906            -0.021590   \n",
       "\n",
       "       NTLmax_denpchV_2015  \n",
       "9910              0.128910  \n",
       "2177             -0.092450  \n",
       "8057             -0.329074  \n",
       "7                -0.088493  \n",
       "18503            -0.319127  \n",
       "5552              0.228587  \n",
       "10721            -0.312739  \n",
       "12674            -0.096993  \n",
       "13277            -0.063145  \n",
       "4339             -0.105414  \n",
       "\n",
       "[10 rows x 390 columns]"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(ResultsFolder, 'NTLDensity.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8af7bc6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e4e01e37",
   "metadata": {},
   "source": [
    "#### Nighttime Lights per Capita"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "74429c3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "Stats = PlaceNames.copy().merge(NTL, how = 'outer', on='Sett_ID')\n",
    "Stats = Stats.merge(Population, how='left', on='Sett_ID')\n",
    "Sensors = ['D_', 'V_']\n",
    "Methods = ['sum', 'avg', 'max']\n",
    "\n",
    "for year in AllStudyYears:\n",
    "    for Sensor in Sensors:\n",
    "        for agg in Methods:\n",
    "            RawVar = ''.join(['NTL', agg, Sensor, str(year)])\n",
    "            DenomVar = ''.join(['POPsum', str(year)])\n",
    "            NewVar = ''.join(['NTL', agg, '_pop', Sensor, str(year)])\n",
    "            if ((RawVar in Stats.columns) and (DenomVar in Stats.columns)):\n",
    "                Stats[NewVar] = Stats[RawVar] / Stats[DenomVar]\n",
    "            else:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "6c950974",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\3900494648.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\3900494648.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\3900494648.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\3900494648.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\3900494648.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
      "C:\\Users\\grace\\AppData\\Local\\Temp\\ipykernel_25148\\3900494648.py:9: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`\n",
      "  Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n"
     ]
    }
   ],
   "source": [
    "# Change in density\n",
    "for year in AllStudyYears:\n",
    "    for Sensor in Sensors:\n",
    "        for agg in Methods:\n",
    "            RawVar = ''.join(['NTL', agg, '_pop', Sensor, str(year)])\n",
    "            LagVar = ''.join(['NTL', agg, '_pop', Sensor, str(year-1)])\n",
    "            NewVar = ''.join(['NTL', agg, '_poppch', Sensor, str(year)])\n",
    "            if ((RawVar in Stats.columns) and (LagVar in Stats.columns)):\n",
    "                Stats[NewVar] = (Stats[RawVar] - Stats[LagVar]) / Stats[LagVar]\n",
    "            else:\n",
    "                pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "d68f409e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sett_ID',\n",
       " 'SettName',\n",
       " 'NTLctD_1999',\n",
       " 'NTLsumD_1999',\n",
       " 'NTLavgD_1999',\n",
       " 'NTLmaxD_1999',\n",
       " 'NTLminD_1999',\n",
       " 'NTLcfc_D_1999',\n",
       " 'NTLctD_2000',\n",
       " 'NTLsumD_2000',\n",
       " 'NTLavgD_2000',\n",
       " 'NTLmaxD_2000',\n",
       " 'NTLminD_2000',\n",
       " 'NTLcfc_D_2000',\n",
       " 'NTLctD_2001',\n",
       " 'NTLsumD_2001',\n",
       " 'NTLavgD_2001',\n",
       " 'NTLmaxD_2001',\n",
       " 'NTLminD_2001',\n",
       " 'NTLcfc_D_2001',\n",
       " 'NTLctD_2002',\n",
       " 'NTLsumD_2002',\n",
       " 'NTLavgD_2002',\n",
       " 'NTLmaxD_2002',\n",
       " 'NTLminD_2002',\n",
       " 'NTLcfc_D_2002',\n",
       " 'NTLctD_2003',\n",
       " 'NTLsumD_2003',\n",
       " 'NTLavgD_2003',\n",
       " 'NTLmaxD_2003',\n",
       " 'NTLminD_2003',\n",
       " 'NTLcfc_D_2003',\n",
       " 'NTLctD_2004',\n",
       " 'NTLsumD_2004',\n",
       " 'NTLavgD_2004',\n",
       " 'NTLmaxD_2004',\n",
       " 'NTLminD_2004',\n",
       " 'NTLcfc_D_2004',\n",
       " 'NTLctD_2005',\n",
       " 'NTLsumD_2005',\n",
       " 'NTLavgD_2005',\n",
       " 'NTLmaxD_2005',\n",
       " 'NTLminD_2005',\n",
       " 'NTLcfc_D_2005',\n",
       " 'NTLctD_2006',\n",
       " 'NTLsumD_2006',\n",
       " 'NTLavgD_2006',\n",
       " 'NTLmaxD_2006',\n",
       " 'NTLminD_2006',\n",
       " 'NTLcfc_D_2006',\n",
       " 'NTLctD_2007',\n",
       " 'NTLsumD_2007',\n",
       " 'NTLavgD_2007',\n",
       " 'NTLmaxD_2007',\n",
       " 'NTLminD_2007',\n",
       " 'NTLcfc_D_2007',\n",
       " 'NTLctD_2008',\n",
       " 'NTLsumD_2008',\n",
       " 'NTLavgD_2008',\n",
       " 'NTLmaxD_2008',\n",
       " 'NTLminD_2008',\n",
       " 'NTLcfc_D_2008',\n",
       " 'NTLctD_2009',\n",
       " 'NTLsumD_2009',\n",
       " 'NTLavgD_2009',\n",
       " 'NTLmaxD_2009',\n",
       " 'NTLminD_2009',\n",
       " 'NTLcfc_D_2009',\n",
       " 'NTLctD_2010',\n",
       " 'NTLsumD_2010',\n",
       " 'NTLavgD_2010',\n",
       " 'NTLmaxD_2010',\n",
       " 'NTLminD_2010',\n",
       " 'NTLcfc_D_2010',\n",
       " 'NTLctD_2011',\n",
       " 'NTLsumD_2011',\n",
       " 'NTLavgD_2011',\n",
       " 'NTLmaxD_2011',\n",
       " 'NTLminD_2011',\n",
       " 'NTLcfc_D_2011',\n",
       " 'NTLctD_2012',\n",
       " 'NTLsumD_2012',\n",
       " 'NTLavgD_2012',\n",
       " 'NTLmaxD_2012',\n",
       " 'NTLminD_2012',\n",
       " 'NTLcfc_D_2012',\n",
       " 'NTLctD_2013',\n",
       " 'NTLsumD_2013',\n",
       " 'NTLavgD_2013',\n",
       " 'NTLmaxD_2013',\n",
       " 'NTLminD_2013',\n",
       " 'NTLcfc_D_2013',\n",
       " 'NTLctV_2012',\n",
       " 'NTLsumV_2012',\n",
       " 'NTLavgV_2012',\n",
       " 'NTLmaxV_2012',\n",
       " 'NTLminV_2012',\n",
       " 'NTLcfc_V_2012',\n",
       " 'NTLctV_2013',\n",
       " 'NTLsumV_2013',\n",
       " 'NTLavgV_2013',\n",
       " 'NTLmaxV_2013',\n",
       " 'NTLminV_2013',\n",
       " 'NTLcfc_V_2013',\n",
       " 'NTLctV_2014',\n",
       " 'NTLsumV_2014',\n",
       " 'NTLavgV_2014',\n",
       " 'NTLmaxV_2014',\n",
       " 'NTLminV_2014',\n",
       " 'NTLcfc_V_2014',\n",
       " 'NTLctV_2015',\n",
       " 'NTLsumV_2015',\n",
       " 'NTLavgV_2015',\n",
       " 'NTLmaxV_2015',\n",
       " 'NTLminV_2015',\n",
       " 'NTLcfc_V_2015',\n",
       " 'POPct2000',\n",
       " 'POPsum2000',\n",
       " 'POPct2001',\n",
       " 'POPsum2001',\n",
       " 'POPct2002',\n",
       " 'POPsum2002',\n",
       " 'POPct2003',\n",
       " 'POPsum2003',\n",
       " 'POPct2004',\n",
       " 'POPsum2004',\n",
       " 'POPct2005',\n",
       " 'POPsum2005',\n",
       " 'POPct2006',\n",
       " 'POPsum2006',\n",
       " 'POPct2007',\n",
       " 'POPsum2007',\n",
       " 'POPct2008',\n",
       " 'POPsum2008',\n",
       " 'POPct2009',\n",
       " 'POPsum2009',\n",
       " 'POPct2010',\n",
       " 'POPsum2010',\n",
       " 'POPct2011',\n",
       " 'POPsum2011',\n",
       " 'POPct2012',\n",
       " 'POPsum2012',\n",
       " 'POPct2013',\n",
       " 'POPsum2013',\n",
       " 'POPct2014',\n",
       " 'POPsum2014',\n",
       " 'POPct2015',\n",
       " 'POPsum2015',\n",
       " 'NTLsum_popD_2000',\n",
       " 'NTLavg_popD_2000',\n",
       " 'NTLmax_popD_2000',\n",
       " 'NTLsum_popD_2001',\n",
       " 'NTLavg_popD_2001',\n",
       " 'NTLmax_popD_2001',\n",
       " 'NTLsum_popD_2002',\n",
       " 'NTLavg_popD_2002',\n",
       " 'NTLmax_popD_2002',\n",
       " 'NTLsum_popD_2003',\n",
       " 'NTLavg_popD_2003',\n",
       " 'NTLmax_popD_2003',\n",
       " 'NTLsum_popD_2004',\n",
       " 'NTLavg_popD_2004',\n",
       " 'NTLmax_popD_2004',\n",
       " 'NTLsum_popD_2005',\n",
       " 'NTLavg_popD_2005',\n",
       " 'NTLmax_popD_2005',\n",
       " 'NTLsum_popD_2006',\n",
       " 'NTLavg_popD_2006',\n",
       " 'NTLmax_popD_2006',\n",
       " 'NTLsum_popD_2007',\n",
       " 'NTLavg_popD_2007',\n",
       " 'NTLmax_popD_2007',\n",
       " 'NTLsum_popD_2008',\n",
       " 'NTLavg_popD_2008',\n",
       " 'NTLmax_popD_2008',\n",
       " 'NTLsum_popD_2009',\n",
       " 'NTLavg_popD_2009',\n",
       " 'NTLmax_popD_2009',\n",
       " 'NTLsum_popD_2010',\n",
       " 'NTLavg_popD_2010',\n",
       " 'NTLmax_popD_2010',\n",
       " 'NTLsum_popD_2011',\n",
       " 'NTLavg_popD_2011',\n",
       " 'NTLmax_popD_2011',\n",
       " 'NTLsum_popD_2012',\n",
       " 'NTLavg_popD_2012',\n",
       " 'NTLmax_popD_2012',\n",
       " 'NTLsum_popV_2012',\n",
       " 'NTLavg_popV_2012',\n",
       " 'NTLmax_popV_2012',\n",
       " 'NTLsum_popD_2013',\n",
       " 'NTLavg_popD_2013',\n",
       " 'NTLmax_popD_2013',\n",
       " 'NTLsum_popV_2013',\n",
       " 'NTLavg_popV_2013',\n",
       " 'NTLmax_popV_2013',\n",
       " 'NTLsum_popV_2014',\n",
       " 'NTLavg_popV_2014',\n",
       " 'NTLmax_popV_2014',\n",
       " 'NTLsum_popV_2015',\n",
       " 'NTLavg_popV_2015',\n",
       " 'NTLmax_popV_2015',\n",
       " 'NTLsum_poppchD_2001',\n",
       " 'NTLavg_poppchD_2001',\n",
       " 'NTLmax_poppchD_2001',\n",
       " 'NTLsum_poppchD_2002',\n",
       " 'NTLavg_poppchD_2002',\n",
       " 'NTLmax_poppchD_2002',\n",
       " 'NTLsum_poppchD_2003',\n",
       " 'NTLavg_poppchD_2003',\n",
       " 'NTLmax_poppchD_2003',\n",
       " 'NTLsum_poppchD_2004',\n",
       " 'NTLavg_poppchD_2004',\n",
       " 'NTLmax_poppchD_2004',\n",
       " 'NTLsum_poppchD_2005',\n",
       " 'NTLavg_poppchD_2005',\n",
       " 'NTLmax_poppchD_2005',\n",
       " 'NTLsum_poppchD_2006',\n",
       " 'NTLavg_poppchD_2006',\n",
       " 'NTLmax_poppchD_2006',\n",
       " 'NTLsum_poppchD_2007',\n",
       " 'NTLavg_poppchD_2007',\n",
       " 'NTLmax_poppchD_2007',\n",
       " 'NTLsum_poppchD_2008',\n",
       " 'NTLavg_poppchD_2008',\n",
       " 'NTLmax_poppchD_2008',\n",
       " 'NTLsum_poppchD_2009',\n",
       " 'NTLavg_poppchD_2009',\n",
       " 'NTLmax_poppchD_2009',\n",
       " 'NTLsum_poppchD_2010',\n",
       " 'NTLavg_poppchD_2010',\n",
       " 'NTLmax_poppchD_2010',\n",
       " 'NTLsum_poppchD_2011',\n",
       " 'NTLavg_poppchD_2011',\n",
       " 'NTLmax_poppchD_2011',\n",
       " 'NTLsum_poppchD_2012',\n",
       " 'NTLavg_poppchD_2012',\n",
       " 'NTLmax_poppchD_2012',\n",
       " 'NTLsum_poppchD_2013',\n",
       " 'NTLavg_poppchD_2013',\n",
       " 'NTLmax_poppchD_2013',\n",
       " 'NTLsum_poppchV_2013',\n",
       " 'NTLavg_poppchV_2013',\n",
       " 'NTLmax_poppchV_2013',\n",
       " 'NTLsum_poppchV_2014',\n",
       " 'NTLavg_poppchV_2014',\n",
       " 'NTLmax_poppchV_2014',\n",
       " 'NTLsum_poppchV_2015',\n",
       " 'NTLavg_poppchV_2015',\n",
       " 'NTLmax_poppchV_2015']"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "c2dc4c4f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Sett_ID',\n",
       " 'SettName',\n",
       " 'NTLsum_popD_2000',\n",
       " 'NTLavg_popD_2000',\n",
       " 'NTLmax_popD_2000',\n",
       " 'NTLsum_popD_2001',\n",
       " 'NTLavg_popD_2001',\n",
       " 'NTLmax_popD_2001',\n",
       " 'NTLsum_popD_2002',\n",
       " 'NTLavg_popD_2002',\n",
       " 'NTLmax_popD_2002',\n",
       " 'NTLsum_popD_2003',\n",
       " 'NTLavg_popD_2003',\n",
       " 'NTLmax_popD_2003',\n",
       " 'NTLsum_popD_2004',\n",
       " 'NTLavg_popD_2004',\n",
       " 'NTLmax_popD_2004',\n",
       " 'NTLsum_popD_2005',\n",
       " 'NTLavg_popD_2005',\n",
       " 'NTLmax_popD_2005',\n",
       " 'NTLsum_popD_2006',\n",
       " 'NTLavg_popD_2006',\n",
       " 'NTLmax_popD_2006',\n",
       " 'NTLsum_popD_2007',\n",
       " 'NTLavg_popD_2007',\n",
       " 'NTLmax_popD_2007',\n",
       " 'NTLsum_popD_2008',\n",
       " 'NTLavg_popD_2008',\n",
       " 'NTLmax_popD_2008',\n",
       " 'NTLsum_popD_2009',\n",
       " 'NTLavg_popD_2009',\n",
       " 'NTLmax_popD_2009',\n",
       " 'NTLsum_popD_2010',\n",
       " 'NTLavg_popD_2010',\n",
       " 'NTLmax_popD_2010',\n",
       " 'NTLsum_popD_2011',\n",
       " 'NTLavg_popD_2011',\n",
       " 'NTLmax_popD_2011',\n",
       " 'NTLsum_popD_2012',\n",
       " 'NTLavg_popD_2012',\n",
       " 'NTLmax_popD_2012',\n",
       " 'NTLsum_popV_2012',\n",
       " 'NTLavg_popV_2012',\n",
       " 'NTLmax_popV_2012',\n",
       " 'NTLsum_popD_2013',\n",
       " 'NTLavg_popD_2013',\n",
       " 'NTLmax_popD_2013',\n",
       " 'NTLsum_popV_2013',\n",
       " 'NTLavg_popV_2013',\n",
       " 'NTLmax_popV_2013',\n",
       " 'NTLsum_popV_2014',\n",
       " 'NTLavg_popV_2014',\n",
       " 'NTLmax_popV_2014',\n",
       " 'NTLsum_popV_2015',\n",
       " 'NTLavg_popV_2015',\n",
       " 'NTLmax_popV_2015',\n",
       " 'NTLsum_poppchD_2001',\n",
       " 'NTLavg_poppchD_2001',\n",
       " 'NTLmax_poppchD_2001',\n",
       " 'NTLsum_poppchD_2002',\n",
       " 'NTLavg_poppchD_2002',\n",
       " 'NTLmax_poppchD_2002',\n",
       " 'NTLsum_poppchD_2003',\n",
       " 'NTLavg_poppchD_2003',\n",
       " 'NTLmax_poppchD_2003',\n",
       " 'NTLsum_poppchD_2004',\n",
       " 'NTLavg_poppchD_2004',\n",
       " 'NTLmax_poppchD_2004',\n",
       " 'NTLsum_poppchD_2005',\n",
       " 'NTLavg_poppchD_2005',\n",
       " 'NTLmax_poppchD_2005',\n",
       " 'NTLsum_poppchD_2006',\n",
       " 'NTLavg_poppchD_2006',\n",
       " 'NTLmax_poppchD_2006',\n",
       " 'NTLsum_poppchD_2007',\n",
       " 'NTLavg_poppchD_2007',\n",
       " 'NTLmax_poppchD_2007',\n",
       " 'NTLsum_poppchD_2008',\n",
       " 'NTLavg_poppchD_2008',\n",
       " 'NTLmax_poppchD_2008',\n",
       " 'NTLsum_poppchD_2009',\n",
       " 'NTLavg_poppchD_2009',\n",
       " 'NTLmax_poppchD_2009',\n",
       " 'NTLsum_poppchD_2010',\n",
       " 'NTLavg_poppchD_2010',\n",
       " 'NTLmax_poppchD_2010',\n",
       " 'NTLsum_poppchD_2011',\n",
       " 'NTLavg_poppchD_2011',\n",
       " 'NTLmax_poppchD_2011',\n",
       " 'NTLsum_poppchD_2012',\n",
       " 'NTLavg_poppchD_2012',\n",
       " 'NTLmax_poppchD_2012',\n",
       " 'NTLsum_poppchD_2013',\n",
       " 'NTLavg_poppchD_2013',\n",
       " 'NTLmax_poppchD_2013',\n",
       " 'NTLsum_poppchV_2013',\n",
       " 'NTLavg_poppchV_2013',\n",
       " 'NTLmax_poppchV_2013',\n",
       " 'NTLsum_poppchV_2014',\n",
       " 'NTLavg_poppchV_2014',\n",
       " 'NTLmax_poppchV_2014',\n",
       " 'NTLsum_poppchV_2015',\n",
       " 'NTLavg_poppchV_2015',\n",
       " 'NTLmax_poppchV_2015']"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop original variables.\n",
    "Stats = Stats.loc[:, Stats.columns.str.contains('Sett|_pop')]\n",
    "list(Stats.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "dc5e9320",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sett_ID</th>\n",
       "      <th>SettName</th>\n",
       "      <th>year</th>\n",
       "      <th>ADM_ID_x</th>\n",
       "      <th>AREA2015</th>\n",
       "      <th>AREA2014</th>\n",
       "      <th>AREA2013</th>\n",
       "      <th>AREA2012</th>\n",
       "      <th>AREA2011</th>\n",
       "      <th>AREA2010</th>\n",
       "      <th>...</th>\n",
       "      <th>NTLmax_poppchD_2013</th>\n",
       "      <th>NTLsum_poppchV_2013</th>\n",
       "      <th>NTLavg_poppchV_2013</th>\n",
       "      <th>NTLmax_poppchV_2013</th>\n",
       "      <th>NTLsum_poppchV_2014</th>\n",
       "      <th>NTLavg_poppchV_2014</th>\n",
       "      <th>NTLmax_poppchV_2014</th>\n",
       "      <th>NTLsum_poppchV_2015</th>\n",
       "      <th>NTLavg_poppchV_2015</th>\n",
       "      <th>NTLmax_poppchV_2015</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9910</th>\n",
       "      <td>227047</td>\n",
       "      <td>Ouagadougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>147</td>\n",
       "      <td>346.334596</td>\n",
       "      <td>338.023014</td>\n",
       "      <td>326.382551</td>\n",
       "      <td>318.261337</td>\n",
       "      <td>310.990827</td>\n",
       "      <td>304.469890</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.059993</td>\n",
       "      <td>0.039075</td>\n",
       "      <td>-0.015965</td>\n",
       "      <td>-0.094227</td>\n",
       "      <td>-0.040111</td>\n",
       "      <td>-0.075358</td>\n",
       "      <td>0.029877</td>\n",
       "      <td>-0.067135</td>\n",
       "      <td>-0.114833</td>\n",
       "      <td>0.085868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2177</th>\n",
       "      <td>51196</td>\n",
       "      <td>Bobo-Dioulasso</td>\n",
       "      <td>2015</td>\n",
       "      <td>37</td>\n",
       "      <td>88.690864</td>\n",
       "      <td>87.474722</td>\n",
       "      <td>86.498238</td>\n",
       "      <td>84.906460</td>\n",
       "      <td>82.475025</td>\n",
       "      <td>80.240756</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.071694</td>\n",
       "      <td>0.018834</td>\n",
       "      <td>-0.005935</td>\n",
       "      <td>-0.086209</td>\n",
       "      <td>-0.102029</td>\n",
       "      <td>-0.110677</td>\n",
       "      <td>-0.170381</td>\n",
       "      <td>0.060661</td>\n",
       "      <td>0.015041</td>\n",
       "      <td>-0.092106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8057</th>\n",
       "      <td>202965</td>\n",
       "      <td>Koudougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>157</td>\n",
       "      <td>30.684014</td>\n",
       "      <td>30.157103</td>\n",
       "      <td>29.583451</td>\n",
       "      <td>28.301020</td>\n",
       "      <td>27.843798</td>\n",
       "      <td>27.413772</td>\n",
       "      <td>...</td>\n",
       "      <td>0.132137</td>\n",
       "      <td>0.450227</td>\n",
       "      <td>0.287927</td>\n",
       "      <td>1.244467</td>\n",
       "      <td>-0.182008</td>\n",
       "      <td>-0.173052</td>\n",
       "      <td>-0.290183</td>\n",
       "      <td>-0.061132</td>\n",
       "      <td>-0.064546</td>\n",
       "      <td>-0.320721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>Banfora</td>\n",
       "      <td>2015</td>\n",
       "      <td>14</td>\n",
       "      <td>16.495685</td>\n",
       "      <td>16.022316</td>\n",
       "      <td>15.331384</td>\n",
       "      <td>14.787476</td>\n",
       "      <td>14.615806</td>\n",
       "      <td>14.377846</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.083988</td>\n",
       "      <td>0.031721</td>\n",
       "      <td>0.039537</td>\n",
       "      <td>0.043427</td>\n",
       "      <td>-0.144943</td>\n",
       "      <td>-0.170091</td>\n",
       "      <td>-0.222855</td>\n",
       "      <td>0.270607</td>\n",
       "      <td>0.270607</td>\n",
       "      <td>-0.057837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18503</th>\n",
       "      <td>456905</td>\n",
       "      <td>Ouahigouya</td>\n",
       "      <td>2015</td>\n",
       "      <td>219</td>\n",
       "      <td>13.275584</td>\n",
       "      <td>12.025448</td>\n",
       "      <td>10.522054</td>\n",
       "      <td>9.225176</td>\n",
       "      <td>9.060304</td>\n",
       "      <td>8.580136</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.135647</td>\n",
       "      <td>0.023811</td>\n",
       "      <td>-0.025276</td>\n",
       "      <td>-0.067188</td>\n",
       "      <td>-0.025001</td>\n",
       "      <td>-0.051001</td>\n",
       "      <td>-0.003826</td>\n",
       "      <td>-0.116980</td>\n",
       "      <td>-0.122828</td>\n",
       "      <td>-0.242030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5552</th>\n",
       "      <td>137698</td>\n",
       "      <td>Dedougou</td>\n",
       "      <td>2015</td>\n",
       "      <td>64</td>\n",
       "      <td>12.117232</td>\n",
       "      <td>11.673608</td>\n",
       "      <td>11.322618</td>\n",
       "      <td>10.958880</td>\n",
       "      <td>10.829702</td>\n",
       "      <td>10.616388</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017398</td>\n",
       "      <td>-0.066791</td>\n",
       "      <td>-0.076615</td>\n",
       "      <td>-0.496595</td>\n",
       "      <td>0.381558</td>\n",
       "      <td>0.312481</td>\n",
       "      <td>0.101906</td>\n",
       "      <td>0.447337</td>\n",
       "      <td>0.186342</td>\n",
       "      <td>0.103455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10721</th>\n",
       "      <td>247085</td>\n",
       "      <td>Tenkodogo</td>\n",
       "      <td>2015</td>\n",
       "      <td>296</td>\n",
       "      <td>11.164545</td>\n",
       "      <td>11.025168</td>\n",
       "      <td>10.938483</td>\n",
       "      <td>10.821203</td>\n",
       "      <td>10.260299</td>\n",
       "      <td>9.979847</td>\n",
       "      <td>...</td>\n",
       "      <td>0.012896</td>\n",
       "      <td>0.077888</td>\n",
       "      <td>0.068432</td>\n",
       "      <td>-0.066106</td>\n",
       "      <td>-0.123327</td>\n",
       "      <td>-0.180813</td>\n",
       "      <td>-0.276898</td>\n",
       "      <td>-0.134871</td>\n",
       "      <td>-0.120453</td>\n",
       "      <td>-0.346575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12674</th>\n",
       "      <td>309529</td>\n",
       "      <td>Pouytenga</td>\n",
       "      <td>2015</td>\n",
       "      <td>249</td>\n",
       "      <td>10.984375</td>\n",
       "      <td>10.879843</td>\n",
       "      <td>10.762563</td>\n",
       "      <td>10.671629</td>\n",
       "      <td>10.528853</td>\n",
       "      <td>10.370780</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.038226</td>\n",
       "      <td>0.139705</td>\n",
       "      <td>0.105169</td>\n",
       "      <td>0.078358</td>\n",
       "      <td>-0.064250</td>\n",
       "      <td>-0.064250</td>\n",
       "      <td>-0.140872</td>\n",
       "      <td>-0.095317</td>\n",
       "      <td>-0.113231</td>\n",
       "      <td>-0.113646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13277</th>\n",
       "      <td>318632</td>\n",
       "      <td>Fada N'gourma</td>\n",
       "      <td>2015</td>\n",
       "      <td>87</td>\n",
       "      <td>10.748116</td>\n",
       "      <td>10.528003</td>\n",
       "      <td>10.369930</td>\n",
       "      <td>10.135370</td>\n",
       "      <td>9.806476</td>\n",
       "      <td>9.706194</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.023052</td>\n",
       "      <td>0.054648</td>\n",
       "      <td>0.006710</td>\n",
       "      <td>-0.088731</td>\n",
       "      <td>-0.111685</td>\n",
       "      <td>-0.111685</td>\n",
       "      <td>-0.249525</td>\n",
       "      <td>-0.075084</td>\n",
       "      <td>-0.115298</td>\n",
       "      <td>-0.124773</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4339</th>\n",
       "      <td>102549</td>\n",
       "      <td>Hounde</td>\n",
       "      <td>2015</td>\n",
       "      <td>116</td>\n",
       "      <td>9.152938</td>\n",
       "      <td>8.733959</td>\n",
       "      <td>8.262290</td>\n",
       "      <td>7.708185</td>\n",
       "      <td>7.514418</td>\n",
       "      <td>7.334248</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.230847</td>\n",
       "      <td>-0.100396</td>\n",
       "      <td>-0.193002</td>\n",
       "      <td>-0.162385</td>\n",
       "      <td>-0.202730</td>\n",
       "      <td>-0.286653</td>\n",
       "      <td>-0.244450</td>\n",
       "      <td>0.045317</td>\n",
       "      <td>-0.006949</td>\n",
       "      <td>-0.092027</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 492 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Sett_ID        SettName  year  ADM_ID_x    AREA2015    AREA2014  \\\n",
       "9910    227047     Ouagadougou  2015       147  346.334596  338.023014   \n",
       "2177     51196  Bobo-Dioulasso  2015        37   88.690864   87.474722   \n",
       "8057    202965       Koudougou  2015       157   30.684014   30.157103   \n",
       "7            8         Banfora  2015        14   16.495685   16.022316   \n",
       "18503   456905      Ouahigouya  2015       219   13.275584   12.025448   \n",
       "5552    137698        Dedougou  2015        64   12.117232   11.673608   \n",
       "10721   247085       Tenkodogo  2015       296   11.164545   11.025168   \n",
       "12674   309529       Pouytenga  2015       249   10.984375   10.879843   \n",
       "13277   318632   Fada N'gourma  2015        87   10.748116   10.528003   \n",
       "4339    102549          Hounde  2015       116    9.152938    8.733959   \n",
       "\n",
       "         AREA2013    AREA2012    AREA2011    AREA2010  ...  \\\n",
       "9910   326.382551  318.261337  310.990827  304.469890  ...   \n",
       "2177    86.498238   84.906460   82.475025   80.240756  ...   \n",
       "8057    29.583451   28.301020   27.843798   27.413772  ...   \n",
       "7       15.331384   14.787476   14.615806   14.377846  ...   \n",
       "18503   10.522054    9.225176    9.060304    8.580136  ...   \n",
       "5552    11.322618   10.958880   10.829702   10.616388  ...   \n",
       "10721   10.938483   10.821203   10.260299    9.979847  ...   \n",
       "12674   10.762563   10.671629   10.528853   10.370780  ...   \n",
       "13277   10.369930   10.135370    9.806476    9.706194  ...   \n",
       "4339     8.262290    7.708185    7.514418    7.334248  ...   \n",
       "\n",
       "       NTLmax_poppchD_2013  NTLsum_poppchV_2013  NTLavg_poppchV_2013  \\\n",
       "9910             -0.059993             0.039075            -0.015965   \n",
       "2177             -0.071694             0.018834            -0.005935   \n",
       "8057              0.132137             0.450227             0.287927   \n",
       "7                -0.083988             0.031721             0.039537   \n",
       "18503            -0.135647             0.023811            -0.025276   \n",
       "5552              0.017398            -0.066791            -0.076615   \n",
       "10721             0.012896             0.077888             0.068432   \n",
       "12674            -0.038226             0.139705             0.105169   \n",
       "13277            -0.023052             0.054648             0.006710   \n",
       "4339             -0.230847            -0.100396            -0.193002   \n",
       "\n",
       "       NTLmax_poppchV_2013  NTLsum_poppchV_2014  NTLavg_poppchV_2014  \\\n",
       "9910             -0.094227            -0.040111            -0.075358   \n",
       "2177             -0.086209            -0.102029            -0.110677   \n",
       "8057              1.244467            -0.182008            -0.173052   \n",
       "7                 0.043427            -0.144943            -0.170091   \n",
       "18503            -0.067188            -0.025001            -0.051001   \n",
       "5552             -0.496595             0.381558             0.312481   \n",
       "10721            -0.066106            -0.123327            -0.180813   \n",
       "12674             0.078358            -0.064250            -0.064250   \n",
       "13277            -0.088731            -0.111685            -0.111685   \n",
       "4339             -0.162385            -0.202730            -0.286653   \n",
       "\n",
       "       NTLmax_poppchV_2014  NTLsum_poppchV_2015  NTLavg_poppchV_2015  \\\n",
       "9910              0.029877            -0.067135            -0.114833   \n",
       "2177             -0.170381             0.060661             0.015041   \n",
       "8057             -0.290183            -0.061132            -0.064546   \n",
       "7                -0.222855             0.270607             0.270607   \n",
       "18503            -0.003826            -0.116980            -0.122828   \n",
       "5552              0.101906             0.447337             0.186342   \n",
       "10721            -0.276898            -0.134871            -0.120453   \n",
       "12674            -0.140872            -0.095317            -0.113231   \n",
       "13277            -0.249525            -0.075084            -0.115298   \n",
       "4339             -0.244450             0.045317            -0.006949   \n",
       "\n",
       "       NTLmax_poppchV_2015  \n",
       "9910              0.085868  \n",
       "2177             -0.092106  \n",
       "8057             -0.320721  \n",
       "7                -0.057837  \n",
       "18503            -0.242030  \n",
       "5552              0.103455  \n",
       "10721            -0.346575  \n",
       "12674            -0.113646  \n",
       "13277            -0.124773  \n",
       "4339             -0.092027  \n",
       "\n",
       "[10 rows x 492 columns]"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Stats.to_csv(os.path.join(ResultsFolder, 'NTLperCapita.csv'))\n",
    "Stats.drop(columns='SettName', inplace=True)\n",
    "AllStats = AllStats.merge(Stats, how='left', on='Sett_ID')\n",
    "AllStats[AllStats.SettName!='UNK'].sort_values(by=AllStats.columns[5], ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "703ef035",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff9330c0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d914511",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "779bb95d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b85fbd09",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2a58d46",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bceb50f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8da5ce3e",
   "metadata": {},
   "source": [
    "### 10.4 Create cumulative built area rasters for each year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "id": "d8ef4cd2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " [2015, 2014, 2013, 2012, 2011, 2010, 2009, 2008, 2007, 2006, 2005, 2004, 2003, 2002, 2001, 2000, 1999]\n"
     ]
    }
   ],
   "source": [
    "WSFEPath = ''.join([r'Buildup/', 'WSFE_equalarea.tif'])\n",
    "[xsize, ysize, geotransform, geoproj, Z] = readRaster(WSFEPath)\n",
    "WSFERasters = []\n",
    "\n",
    "ReversedStudyYears = []\n",
    "for i in AllStudyYears:\n",
    "    ReversedStudyYears.insert(0,i)\n",
    "print('\\n\\n', ReversedStudyYears)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "id": "23750d4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Finished reclassifying WSFE 2015. Sun Feb 19 18:01:42 2023\n",
      "Finished reclassifying WSFE 2014. Sun Feb 19 18:02:11 2023\n",
      "Finished reclassifying WSFE 2013. Sun Feb 19 18:02:33 2023\n",
      "Finished reclassifying WSFE 2012. Sun Feb 19 18:03:02 2023\n",
      "Finished reclassifying WSFE 2011. Sun Feb 19 18:03:25 2023\n",
      "Finished reclassifying WSFE 2010. Sun Feb 19 18:03:47 2023\n",
      "Finished reclassifying WSFE 2009. Sun Feb 19 18:04:13 2023\n",
      "Finished reclassifying WSFE 2008. Sun Feb 19 18:04:35 2023\n",
      "Finished reclassifying WSFE 2007. Sun Feb 19 18:05:00 2023\n",
      "Finished reclassifying WSFE 2006. Sun Feb 19 18:05:25 2023\n",
      "Finished reclassifying WSFE 2005. Sun Feb 19 18:05:49 2023\n",
      "Finished reclassifying WSFE 2004. Sun Feb 19 18:06:17 2023\n",
      "Finished reclassifying WSFE 2003. Sun Feb 19 18:06:42 2023\n",
      "Finished reclassifying WSFE 2002. Sun Feb 19 18:07:08 2023\n",
      "Finished reclassifying WSFE 2001. Sun Feb 19 18:07:34 2023\n",
      "Finished reclassifying WSFE 2000. Sun Feb 19 18:08:01 2023\n",
      "Finished reclassifying WSFE 1999. Sun Feb 19 18:08:22 2023\n",
      "\n",
      "Single cumulative year WSFE set: ['WSFE_2015.tif', 'WSFE_2014.tif', 'WSFE_2013.tif', 'WSFE_2012.tif', 'WSFE_2011.tif', 'WSFE_2010.tif', 'WSFE_2009.tif', 'WSFE_2008.tif', 'WSFE_2007.tif', 'WSFE_2006.tif', 'WSFE_2005.tif', 'WSFE_2004.tif', 'WSFE_2003.tif', 'WSFE_2002.tif', 'WSFE_2001.tif', 'WSFE_2000.tif', 'WSFE_1999.tif']\n"
     ]
    }
   ],
   "source": [
    "for Year in ReversedStudyYears:\n",
    "    OutPath = ''.join([r'Buildup/ByCuYear/', 'WSFE_', str(Year), '.tif'])\n",
    "    \n",
    "    Z[Z>Year] = 0 # Since we've sorted years by descending order, we don't need to reload the original raster values each time.\n",
    "    \n",
    "    writeRaster(OutPath,geotransform,geoproj,Z)\n",
    "    OutPath = None\n",
    "    WSFERasters = WSFERasters + [''.join(['WSFE_', str(Year), '.tif'])]\n",
    "    \n",
    "    print('Finished reclassifying WSFE %s. %s' % (Year, time.ctime()))\n",
    "\n",
    "print('\\nSingle cumulative year WSFE set: %s' % WSFERasters)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}