Quantify uncertainty using conformal prediction for classification and regression tasks in Google Earth Engine. Examples for 1) tree canopy height estimation using conformal quantile regression using Planet NICFI and GEDI data, 2) uncertainty quantification for Google Dynamic World landcover data and 3) for invasive tree species mapping.
Run through an end-to-end example»
View More Examples
·
Report Bug
·
Request Feature
Table of Contents
This Github repository contains the code for the arxiv, peer-reviewed and ICLR 24 short version research paper "Uncertainty quantification for probabilistic machine learning in earth observation using conformal prediction". It includes functionality to quantify uncertainty using conformal prediction, create gap filled cloud-free image composites for multiple time steps and spatial cross validation.
For a more gentle introduction and example use of conformal prediction for Dynamic world refer to this medium blogpost
Uncertainty Quantification (UQ) provides information on prediction quality and can allow for comparisons and itegration of datasets. Conformal prediction is currently the only UQ framework that can provide pixel wise uncertainty information with valid coverage (i.e., if a 0.9 confidence level is specified, alpha = 0.1, the prediction regions will contain the actual value with a 90% probability).
Features
- GEE-Native Conformal classifier and regressor support
- Support for GEE-JS and GEE-Python API
- Create cloud-free image composites of Sentinel-2 and gap-free composites for Sentinel-1.
- Examples to demonstrate end-to-end workflows that include, visualisation, API-based imports and exports, coomputation of additional topographic and coordinate-transformed variables, model fitting, and inference.
- Spatial cross-validation.
In its current state, this repo is meant to be used from Google colab. As such the functionality can be used by cloning this repository. Refer to the linked example above for a guided demonstration.
For the end-to-end example provided, the reference points (shapefile) are already uploaded to Google Earth Engine (GEE). However, if you need to upload your own shapefile to GEE you could either manually upload it or you could upload it by using the geeup package. An example using the geeup package can be found here.
The repository with all GEE conformal JS code can be accessed using this link: https://code.earthengine.google.com/?accept_repo=users/geethensingh/conformal
The repo contains code to calibrate, evaluate and run inference using the calibrated conformal classifier or regressor for both feature and imge collections. Demos are provided for each script.
Assuming a model has already been trained for a classification or regression task and there is an out-of-sample dataset available for calibrating and evaluating a conformal predictor, there are three steps to quantify uncertainty with a guranteed coverage (as shown below).
Note: There are distinct modules for ImageCollections and FeatureCollections. The inference module is the only shared module.
Step
- Calibrate a conformal predictor. Example usuage
//import conformal classifier calibration functions
var calFunctions = require('users/geethensingh/conformal:calibrateConformalFeatureClassifier.js');
// Configuration parameters
var ALPHA = 0.1; // 1-ALPHA corresponds to required coverage. For example, 0.1 for 90% coverage
var SCALE = 10; // Used to compute Eval metrics
var SPLIT = 0.8; // Split used for calibration and test data (for example, 0.8 corresponds to 80% calibration data)
var LABEL = 'label'; //band name for reference label band
// points refer to an out-of-sample featurecollection (not used during training), that contains, as properties,
1) the pseudo-probability of each candidate class
2) the reference label (called 'label' in this case). It should be zero-indexed
3) a value from the randomColumn function used to split the data into a calibration and evaluation dataset based on the user specified SPLIT
var result = calFunctions.calibrate(points, bands, ALPHA, SCALE, SPLIT, LABEL, 'demoDW_15112023');
print(result);//returns qhat value to be used during evaluation and inference (uncertainty quantification)- Evaluate conformal predictor. Example usuage
//import conformal classifier evaluation functions
var evalFunctions = require('users/geethensingh/conformal:evaluateConformalFeatureClassifier.js');
// Configuration parameters
var QHAT = 0.06450596045364033; // from calibration script
var SCALE = 10; // Used to compute Eval metrics
var SPLIT = 0.8; // Split used for calibration and test data (for example, 0.8 corresponds to 80% calibration data, 20% evaluation data)
// points refer to the out-of-sample featurecollection (not used during training but the same collection provided in the calibration stage), that contains, as properties,
1) the pseudo-probability of each candidate class
2) the reference label (called 'label' in this case). It should be zero-indexed
3) a value from the randomColumn function used to split the data into a calibration and evaluation dataset based on the user specified SPLIT
var result = evalFunctions.evaluate(points, bands, QHAT, SCALE, SPLIT, 'demoDW_15112023');
print(result);// This will return the empirical coverage and the average set size- Quantify uncertainty for a test image Example usuage
// Import conformal classifier inference functions
var infFunctions = require('users/geethensingh/conformal:inferenceConformalImageClassifier.js');
// Configuration parameters
var QHAT = 0.06067845112312009; // From calibration script
// Uncertainty Quantification: inference function is mapped over every dynamic world image in the collection
//dwp refers to an ee.Image with a probability band for each candidate band. if using an ee.Classifier, use arrayFlatten([[bandNames]])
var result = infFunctions.inference(dwp, bands, QHAT);The GEE-Python conformal classifiers can be used in a similar fashion to the JavaScript modules and must be loaded by cloning the github repo (as shown below)
# Install and import packages
%pip install watermark geemap geeml -q
!git clone https://github.com/Geethen/Invasive_Species_Mapping.git
import sys
sys.path.insert(0,'/content/Invasive_Species_Mapping/code')
# Authenticate GEE
ee.Authenticate()
# Initialize GEE with high-volume end-point
ee.Initialize(opt_url='https://earthengine-highvolume.googleapis.com')For more examples (Includes three case studies from the research paper), please refer to the Example usuage folder
- Add support for RAPS and APS conformal prediction
- Add support for cloud and shadow masking using CloudScore+
- Add support for mondrian conformal prediction
- Add more examples
See the open issues for a full list of proposed features (and known issues).
Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License. See LICENSE.txt for more information.
Glenn Moncrieff - @Glennwithtwons
Geethen Singh - @Geethen - [email protected]
The Invasive Species Mapping code provided here has been ported from code originally written by Glenn Moncrieff in GEE JavaScript.